Some people are threatened by the "different" because "different" can in many cases mean "deadly".
If we only look at (for example) one society--such as 21st century United States citizenry--the concept may not seem all that clear. But if we take the long view, look back over the millennia of Man's sojourn on this planet, it's as clear as the nose on my wife's face before her nose job.
Let's say you were an Arab during the time of the Crusades and saw the "different" armor of a Christian war column advancing on your village...or you were a Lakota warrior who saw the "different" bluecoats of George Armstrong Custer's 7th Cavalry charging your encampment on the Little Big Horn...or you were an African man seeing the "different" slave ships of the white man come to take you away forever through the Door of No Return...then, and in a million similar situations, you would learn (if you did not know already) that DIFFERENT was only another word for LOOK OUT!!!!
In other words, the very survival of homo sapiens as a species has long depended on being very, very wary when something different showed up. Different was deadly dangerous until proven otherwise.
Personally, I still feel that way about aliens from other planets. If an emissary from beyond the solar system drops by offering gifts, but he/she/it happens to resemble a 27-legged spider with a T-rex head and a rattlesnake tail, I'm going to consider it deadly until proven otherwise.
Do some of us take that fear too far, to the point that the fear owns us instead of serving us by enhancing our chances for survival? Of course we do. It's human nature.