This generalizations stands in perfect opposition to all of my own experience with every large family with which I have had contact. Perhaps it transcends mere numbers. I would suggest it depends on generation, culture, religion, educational expectations of parents and education level of parents.
As a product of 1960's and 70's Catholic education, I have known families from all races, many different countries and of all sizes. I suspect it has far more to do with material expectations, societal and cultural influences and parental expectations than number of children.
This "all swans are white" type generalization is a distortion that does not lead to understanding. A college education is hardly a guarantor of moral superiority or a meaningful and successful life. I know several people who were born and raised in large families who have never experienced the intellectual privations in your stereotype, on the contrary, the children of those families are ALL college educated or gainfully employed in skilled trades. They are educators, business owners, medical professionals, electricians, plumbers, sheet metal workers, civil servants, etc... The previous generation also came from large families and were college educated, skilled tradesmen or civil servants. All were intact families with parents who remained married their entire lives and married after completing secondary school and before having children.
I am certain that merely relying on the number of children and ignoring the tapestry of human experience leads to poor social science.