Google's Blacklisting of Websites — The Dirty Little Secret of Web Searching

Let's say you're doing some heavy research—for a school paper or PhD dissertation, a professional paper, an investigative article, or perhaps an exotic disease afflicting a close friend or relative ... whatever. You need a really in-depth, comprehensive Internet search, right? And you can rely on Google to carefully and thoroughly search the Web and provide you with an exhaustive and tediously comprehensive listing of everything the Internet has to offer on your topic—right?

Unfortunately, probably not.

Google appears to have changed. It's no longer the neutral, unbiased, highly utilitarian tool we all used to know and rely on for those exhaustive, tedious searches to produce that comprehensive listing of search results. Google now thinks, it now has opinions, and it now is making judgments about what you get to search and view.

According to Google Support, "Google may temporarily or permanently remove sites from its index and search results if it believes it is obligated to do so by law, if the sites do not meet Google's quality guidelines, or for other reasons, such as if the sites detract from users' ability to locate relevant information."

Translation: You're gonna get to see what Google wants you to see.

Google's foray into censorship

Within the last year or so, this program of screening search material (websites) and filtering the search results (SERPs, or Search Engine Results Pages) has accelerated and intensified. Part of this is the so-called Panda algorithm, which penalizes websites for lack of "fresh" content, and for what Google's Panda robot deems excessively duplicative content. (I've reported on this in my article Quality Articles Fall Victim to Google Panda and Anti-Duplication Hysteria.)

What Panda does is condemn websites to near-oblivion — whereas a site (say, the source of your bread-and-butter business income) may have ranked fairly high a year or so ago in searches for your product or service, after Panda kicked in early in 2011, many site owners found their pages would rank down around page 300 or so in the SERPs (search results). Likewise, if your webpage held one or more unique bits of information, or perhaps collected information on a topic in a unique way, you might well find that, post-Panda, it would be relegated in SERPs to the far reaches of the Web universe so that just about nobody would ever see it. Basically, a lot of far less relevant results, plus junk, would be inserted in front of it, thanks to Panda.

But those misfortunes pale in comparison to Google's blacklisting binge.

11 million sites blacklisted

Systematically, Google has been placing "undesirable" websites on a massive blacklist, and has been apparently ratcheting up this draconian de facto censorship over approximately the last year. So that, today, an astounding (brace yourself) 11 million — that's ELEVEN MILLION — websites are now on Google's blacklist. That's 11 million sites that users relying on Google to search for stuff will never, ever see.

"Over 11 million websites have been blacklisted and banned by Google" recounts Real News Reporter (20 July 2011). This includes "A massive block of websites registered for free through the co.cc subdomain [which] have been silenced after the Internet giants have determined many of them to be unfit for the Web."

Now, to be fair, among the 11 million sites on the blacklist, some do have problems — some contain malware (malicious software, like viruses), some violate Google's aversion to excessive duplication, some have tried too hard to finesse Search Engine Optimization (SEO), a technique to supposedly improve a site's chances of raising its SERP ranking — and a technique which Google apparently has decided it really hates.

However, Google's blacklisting also has a strongly arbitrary character. File-sharing sites such as The Pirate Bay, isoHunt, 4Shared, and others have been blacklisted because ... well, Google just doesn't like file-sharing? According to the Geeky-Gadgets website (24 November 2011), "It's thought that Google has implemented the new changes to stay on the right side of major movie houses and music studios."

The blacklist also seems to have a political bias as well. In a 30 November 2010 story, Infowars.com reported ominously that "In a damning new lurch towards web censorship, Google’s news aggregator has blacklisted Prison Planet and Infowars despite the fact that both websites are internationally known and now attract more traffic than many mainstream media websites, while Google-owned You Tube has frozen the Alex Jones Channel based on a spurious complaint about showing Wikileaks footage that has been carried on hundreds of other You Tube channels for months."

NOTE: Since the original publication of this article, additional information suggests that that some of these sites can now be found in some types of Google searches. First, Google apparently modified its blacklist to filter certain “piracy-related” terms only from its Autocomplete and Instant services. In addition, anti-censorship activists have developed software, including plug-ins for popular browsers such as Mozilla Firefox, that essentially thwart Google's blacklisting efforts. For more information, see:

Google Now Censors The Pirate Bay, isoHunt, 4Shared and More

Google Censorship Initiative Thwarted by ‘Gee! No Evil!’ Add-On

These developments do not alter the main contention of this article focusing on Google's newly escalated censoring and blacklisting program.

Arbiter of quality

The murky and arbitrary nature of the blacklist is suggested in statements by company executives. For example, as Real News Reporter notes, Google’s Matt Cutts, writing on his Google Plus account, argued that the company has the right to zap sites if they “see a very large fraction of sites on a specific freehost be spammy or low-quality.”

Spammy or low-quality? What does that mean? And that seems to imply an awfully broad and arbitrary brush. What if you were denied service at a restaurant because they refused to serve "gunslingers and people of low quality"? Do you figure you're in the latter category? Moreover, they won't tell you what "quality" rule you violate, nor why. And, oh yeah ... this is a restaurant in Death Valley, the only one with food or water for 100 miles, so you don't really have a viable alternative.

Google's behavior has begun to prompt legal counterattacks, at least in Europe. As described in an article last summer in Search Engine Land (28 June 2011), several companies (including Foundem and Microsoft-owned Ciao) filed formal antitrust complaints against Google with the European Commission in 2010. "Those complaints led directly to the EC’s decision to pursue an antitrust action against Google, which is now in process" reports the website.

Alternatives to Google's power?

Legal fights aside, for most users, Google's blacklist — and Panda — have produced nothing short of a crisis. Google now dominates the online search market — 70 to 80 percent of searches in the USA rely on Google, and an even higher percentage in Europe — 90 percent. Google has reshaped the business world, since businesses now depend on their Internet presence, and in turn rely on Google to help prospective customers find them.

In effect, Google has become a huge pipe, or conduit, through which almost all business marketing, and the vast majority of all informational searching, must be channeled. But now, with its blacklisting of entire sites, and mysterious efforts at manipulation of content via Panda, Google has started down a slippery slope into censoring and asserting control over what can be found on the Web.

Do consumers have any options? Well, there are alternative search engines — Microsoft's Bing.com, for example, which seems to produce even more comprehensive SERPs without a blacklist (at least currently). There's also the possibility of government regulation (which could ultimately prove even more dangerous).

The immediate prospect here doesn't look nice. What it could mean is that, if you're searching for effective therapies or cures for that rare disease afflicting your relative, and the exact solution you need happens to be described on a website bumped to page 500 in the search results, or—even worse—placed on the blacklist, because it fails to meet Google's "quality" standards ... Google might expect you to start searching for a funeral parlor, instead. Maybe one that meets its "quality" standards?