Because of a lack of understanding. If you see someone with whom you share a variety of key traits (eduction level, income level, religion, ect.), you feel as if you have some understanding of what drives them as a person, and how they will act. When you see someone with whom you have almost nothing in common, you don't feel as though you can understand them on almost any level, and things that you cannot understand are feared.
Example: I'm an atheist. For a Christian who has grown up with the idea that the only stable moral code comes from God, I'm dangerous. Clearly, nothing constrains me from doing whatever I want anytime that I want to do it. The most extreme people really don't understand what stops me from going out and cheating, lying, scamming, sleeping around, and committing all forms of murder and mayhem because, according to them, I don't have a moral code to stop me. Or, if I do, it is a moral code of convenience.
The reality is quite different. In actuality, a majority of the moral code I operate under coincides with typical Judeo-Christain values. I'm not a hedonist, which is an apt description of the view that I gave in the previous paragraph. But because I am different, everything that I do is ascribed to shady morals or misconstrued, and they'll avoid me because I'm not trustworthy, which merely perpetuates the cycle.
(Editor's Note: I've met a great many Christians who don't react this way, and who couldn't care less about what I do or do not believe. The example that I've drawn is on the extreme end of how people have responded to me in the past.)