I was an only child also raised by my mother alone (and her parents). To be honest I never really saw or experienced prejudice that suggested we were not a "family" because I was an only child.
I do think there are a lot of assumptions made about only children, that we are selfish, spoiled, pampered and don't know how to share or "play/work well with others". There can be societal, religious and cultural pressure on couples to have more than one child. But I have not personally heard or seen the suggestion that you are not a family if you only have one child.
I DO, however, experience prejudice regularly today being in a relationship with someone and us having NO children. The idea that you are NOT a "family" until you have a child (or more than one) is everywhere today. People asking "When are you going to start a family" or celebrating couples who announce they are "so happy to be starting our family" once they are expecting, etc. Now that's something I resent significantly and wish would change. It's very much possible to have a family of two, just like some might have a family of three, four, five or more.