No it is not fair. The rich today get to decide everything because of a long history of the evolution of class division. When humanity perfected production beyond hunting and gathering subsistence and created a surplus, this allowed for some to break off and to do different things than hunting - gathering and agriculture. As specialization intensified it grew to include defense, religion - astronomy, the trades and everything else we now know. From primitive societies of tribes, to theocracy, to monarchy and finally the rule of the wealthy, it has been the result of alienation between people and the bullying of many by a few who initially took the surplus by force. This process eventually became enshrined by law.
If the poor, the real producers were to "suddenly disappear" then the foundation of production and real wealth creation would disappear too. But; as we now have robots to do much of the heavy, hard, dirty and dangerous work, many producers are no longer required. These traditionally get thrown out of the production process with each advancement or crisis of overproduction. Thus, if the producers who are now "unnecessary" due to the existence of the robot, disappeared, the wealthy would not miss them as machines would keep production and thus real wealth flowing.
The wealthy also control the media, which is why the poor have almost no voice. They are shamed as being lazy when put out of hard labor due to the fact of being replaced by machines. It is this accusation the prevents interventions such social safety nets from being effective and in some cases, terminated. When forced into absolute desperation, many poor turn to crime and prostitution. Remember, crime id defined by law. A poor person stealing a pizza slice is guilty of a felony that could land them in prison. A wealthy person erecting a Ponzi scheme and embezzling billions is protected by the very laws and economic safety valves designed to keep the poor out and the wealthy bailed out.