I would like to believe that there is an evolutionary basis to this question.
Back when man was still trying to figure out on to survive on this earth, the "pack mentality" ruled. There was strength to be found in numbers. This concept can be seen in primitive hunting parties, which consisted of several of the most elite members of the group.
Even during this time and with the pack mentality in place, people who were considered to be different and who could not contribute to the overall survival of the group were cast out. These individuals might include members with mental illness, physical handicap, or disease. Oftentimes they were forced to leave because their presence became a handicap rather than an asset.
Simply put, it you were different and alone during these times, you were nothing. Something would eventually eat you and everybody else knew it.
I believe that even in these great technological times, we as animals still retain many of our primitive attitudes despite the fact that they no longer apply. There is still an undercurrent of belief that there is something wrong with people who remain alone for lengthy periods during their lifetimes because it was at one point considered unnatural. We label one who chooses isolation over socialization as being "different" and then we proceed to ostracize them as we've done for thousands of years. I for one don't believe this philosophy to be true, and in fact I think it's just another basis on which people can negatively judge others.
So, in a nutshell, if you're happy with your own solitude, then our primitive-minded society can go kick rocks.