Adultery would be considered worse in that a married couple has exchanged vows to "forsake all others". Anyone having sex with someone other than their spouse under those vows is guilty of betraying their mate.
Polygamy on the other hand is nothing more than another lifestyle each party has "agreed to". A marriage is entered into with the (knowledge, understaing, and acceptance) that one person may have multiple marital partners.
Life is a personal journey and each of us gets to choose how we want to live our life.