It's only a double standard if you also believe that allowing heterosexual marriage, while banning polygamy, is unfair. Why you ask?
The core of marriage, the absolute defining feature of the institution, is about two people who love each other expressing a life-long commitment to have and to hold, ti'll "death do they part." There is nothing inherent about polygamy that would exclude people from engaging in committed, loving relationships of this type, the same way heterosexuals do. I think it's ridiculous to ban polygamy. If consenting adults desire to be in the relationship, then I have no beef. It's a free choice, even if I personally would never choose such a situation.
That being said, it's clear that gay marriage and polygamy have no relation whatsoever. To my knowledge, most polygamist relationships have been of the polygyny type, which is entirely heterosexual. I don't believe there is even a word to express the idea of a gay man having multiple husbands, or a gay woman having multiple wives. This shows how language structures our reality.
Banning two, or three, or even four people who are in a committed relationship from marrying is just simple discrimination. There is no alternative way to put it, without engaging in Orwellian doublespeak, which many social conservatives typically do when attempting to justify their bigotry.