Part of the problem is self worth. People with high self esteem are less likely to judge people- that doesn't mean they don't but it is less likely. Judging for a lot of people is a way of making them feel more secure or just worth something. If they are better than others then atleast they are worth something.
But fear of change and that which is different is a natural response. Forgive me for simplifing it to this degree- but its like trying a new food. It may be something you will love but you are intent to stick with what you know.
Prejudice isnot only an issue of color or race. The person (black or white or anything) wearing a suit and working hard is likely to judge the person in the bank dressed in torn overalls and smelling like pigs. Everyone has a nitch they are comfortable with and makes it difficult for them to step pass those feelings. Some people have adverse reaction because the person before them reminds them of some one or many someones who wronged them. For instance there are many people who can not release the feeling of hate and fear of anyone who looks Arab after 9/11. Those who lost a loved one, who were saved before the collapse or who worked at ground zero are the most likely to wrestle with such intense emotion. These people deserve patience.
And of course yes some people are just stupid jerks. That can be found in every race, culture, and status.
The truth is there are lots of reasons people judge. Sometimes being patience and trying to decern their reasons is not a bad idea. Then, ofcourse if they are just jerks atleast you know you are judging them based on personal observation of them and not being prejudice yourself. Why risk lowering yourself to something you hate?