Why do you assume that everyone thinks every Moslem is a terrorist. As to why people feel uncomfortable, well that is due to two major behaviors of Moslems. First, Islam is a very closed religion. Like Judaism or Mormanism, Islam has a very us vs them/true believers vs unbelievers system. Second, non-Moslems see no denunciation of suicide bombers or other barbaric attacks by much of anyone in the Islamic world. After 9/11, for example, people spontaneously started dancing in the streets after the towers came down. That shows a tacit approval of the terrorists. People remember that kind of thing. If you don't denounce barbarism, then you become associated with it.