Well if you have been part of a community, society or a group of people where being racist is a common thing of the day then it will eventually start growing on you.
Also depends on how young you are, if you are born in such an environment then you will easily become like others around you as you are so impressionable. It would help if your parents or your elderly or someone you admire shows you the right path and helps you be compassionate without being influenced by the prevailing opinion.
If you think about it then it is an individual's choice at end of the day. No matter how bad things are going on, our innate wisdom will always show us right from wrong.