One word: Xenophobia. Yoga is not a religion, it is a system of exercises. The practice of Yoga is not an act of worship any more than doing pushups or running laps is an act of worship. (Though the way some exercise freaks worship their own bodies, perhaps this isn't always the case...:) )
Some folks will of course try to tell you that you shouldn't be doing yoga because it will necessarily lead to you abandoning your faith (whatever that faith may be). It won't. Of course, the people telling you this may genuinely be concerned for your welfare, figuring that since they're so easily led astray, you must be equally vulnerable. Or, they may just be interested in controlling you, making sure that you don't do anything they wouldn't approve of.
My advice: don't sweat it. If you find, after taking a yoga class, that /your/ faith /is/ wavering, maybe you should reexamine both yoga and your faith. If you conclude that yoga is a threat to your faith, stop doing it. Simple.