I feel like the deepest reason isn't being judged negatively as much as simply being judged. It is fear of the unknown - as all fear is - that really drives people away from public speaking. You have NO idea how the audience will react yo what you are about to say, and it is scary to think of the many different ways they could react to what you are about to say.
Think of it this way. When you at a friend's house with ten of your closest buddies, you have no problem taking the floor and being outrageous. Why? Because you know that your friends will enjoy what you are saying, there is no unknown in the audience to fear. The result is known, thus the unknown anticipation is not there. People simply sike themselves out because of the unknown factors.