From the physical side of things, presumably because yoga involves moves that require a great deal of flexibility. The wrong clothing would hinder a participant's ability to perform the moves correctly, would be uncomfortable and might even break. Yoga is also supposed to be a relaxing activity. It's a lot easier to be relaxed when you're comfortable than when you're thinking about how your pants are too tight, or your underwear is chafing.
Of course, I don't do yoga, so the above is all speculation.