I don't think people look down on those who ride bikes or mopeds so much as they are irritated -- if they drive motor vehicles. Bicycles especially, but also mopeds -- tend to have a limited speed, holding up traffic because it can be hard to get around them depending on how heavy traffic is and what kind of terrain one is in.
People who ride bicycles tend to ignore traffic laws either because they do not know them or because they think they should be able to ignore them if they want to. Bicyclists are supposed to obey the same traffic laws as other vehicles on the road but often do not.
I spend 6 hours and sometimes more on the road everyday for 9 months of the year. I'm amazed that there aren't more accidents given the terrible driving habits of most people regardless of what kind of vehicle they are driving.
When a bicyclist is pedaling up a long steep hill in the country, possibly a heavily traveled road despite it's narrowness with no shoulders, they are holding up traffic. A vehicle takes a big chance of a head-on collision if they choose to go into the oncoming lane to pass the bicyclist and the bicyclist simply cannot pedal only so fast in any situation. Many people on the road have schedules they must keep for various reasons, and so these bicyclists are holding them up.
Yes, bicyclists have the same rights to be on any road as anyone else, unless forbidden as in the case of freeways, but I'm sure you have been in situations where other people were preventing you from getting something done, so that should give you an idea of how irritating it can be if you are under a tight schedule and someone is in the way.
No doubt some people do look down on bicyclists because they may be viewed as poor, but those people who do that may one day find themselves in that situation and realize that being poor is not a character flaw.
Mainly, I believe it is the inconvenience of slow vehicles on the road that is the irritant. Lots of well off people in fancy neighborhoods ride bicycles and walk for exercise. Former Mayor Michael Bloomberg, a billionaire, used to ride the bus to work.
In big cities people usually take the bus, they do not ride a bicycle because it is unsafe. Not just unsafe due to traffic, but because it is easy to mug someone on a bike compared to in a locked car with the windows up. Hard to carry bags of groceries or other on a bike also. No protection from the weather, but mostly no protection from people and stray animals.