ArtsAutosBooksBusinessEducationEntertainmentFamilyFashionFoodGamesGenderHealthHolidaysHomeHubPagesPersonal FinancePetsPoliticsReligionSportsTechnologyTravel
  • »
  • Technology»
  • Internet & the Web»
  • Search Engines

Google's Blacklisting of Websites — The Dirty Little Secret of Web Searching

Updated on December 26, 2011


Let's say you're doing some heavy research — for a school paper or PhD dissertation, or a professional paper, or an investigative article, or perhaps an exotic disease afflicting a close friend or relative ... whatever. You need a really in-depth, comprehensive Internet search, right? And you can rely on Google to carefully and thoroughly search the Web and provide you with an exhaustive and tediously comprehensive listing of everything the Internet has to offer on your topic — right?

Wrong.

Google has changed. It's no longer the neutral, unbiased, highly utilitarian tool we all used to know and rely on for those exhaustive, tedious searches to produce that comprehensive listing of search results. Google now thinks, it now has opinions, and it now is making judgments about what you get to search and view.

According to Google Support, "Google may temporarily or permanently remove sites from its index and search results if it believes it is obligated to do so by law, if the sites do not meet Google's quality guidelines, or for other reasons, such as if the sites detract from users' ability to locate relevant information."

Translation: You're gonna get to see what Google wants you to see.

Google's foray into censorship

Within the last year or so, this program of screening search material (websites) and filtering the search results (SERPs, or Search Engine Results Pages) has accelerated and intensified. Part of this is the so-called Panda algorithm, which penalizes websites for lack of "fresh" content, and for what Google's Panda robot deems excessively duplicative content. (See my report on this, Quality Articles Fall Victim to Google Panda and Anti-Duplication Hysteria.)

What Panda does is condemn websites to near-oblivion — whereas a site (say, the source of your bread-and-butter business income) may have ranked fairly high a year or so ago in searches for your product or service, after Panda kicked in early in 2011, many site owners found their pages would rank down around page 300 or so in the SERPs (search results). Likewise, if your webpage held one or more unique bits of information, or perhaps collected information on a topic in a unique way, you might well find that, post-Panda, it would be relegated in SERPs to the far reaches of the Web universe so that just about nobody would ever see it. Basically, a lot of far less relevant results, plus junk, would be inserted in front of it, thanks to Panda.

But those misfortunes pale in comparison to Google's blacklisting binge.

11 million sites blacklisted

Systematically, Google has been placing "undesirable" websites on a massive blacklist, and has been apparently ratcheting up this draconian de facto censorship over approximately the last year. So that, today, an astounding (brace yourself) 11 million — that's ELEVEN MILLION — websites are now on Google's blacklist. That's 11 million sites that users relying on Google to search for stuff will never, ever see.

"Over 11 million websites have been blacklisted and banned by Google" recounts Real News Reporter (20 July 2011). This includes "A massive block of websites registered for free through the co.cc subdomain [which] have been silenced after the Internet giants have determined many of them to be unfit for the Web."

Now, to be fair, among the 11 million sites on the blacklist, some do have problems — some contain malware (malicious software, like viruses), some violate Google's aversion to excessive duplication, some have tried too hard to finesse Search Engine Optimization (SEO), a technique to supposedly improve a site's chances of raising its SERP ranking — and a technique which Google apparently has decided it really hates.

However, Google's blacklisting also has a strongly arbitrary character. File-sharing sites such as The Pirate Bay, isoHunt, 4Shared, and others have been blacklisted because ... well, Google just doesn't like file-sharing? According to the Geeky-Gadgets website (24 November 2011), "It's thought that Google has implemented the new changes to stay on the right side of major movie houses and music studios."

The blacklist also seems to have a political bias as well. In a 30 November 2010 story, Infowars.com reported ominously that "In a damning new lurch towards web censorship, Google’s news aggregator has blacklisted Prison Planet and Infowars despite the fact that both websites are internationally known and now attract more traffic than many mainstream media websites, while Google-owned You Tube has frozen the Alex Jones Channel based on a spurious complaint about showing Wikileaks footage that has been carried on hundreds of other You Tube channels for months."

NOTE: Since the original publication of this article, additional information on some the websites mentioned above has emerged. It appears that some of these sites can now be found in some types of Google searches. First, Google apparently modified its blacklist to filter certain “piracy-related” terms only from its Autocomplete and Instant services. In addition, anti-censorship activists have developed software, including plug-ins for popular browsers such as Mozilla Firefox, that essentially thwart Google's blacklisting efforts. For more information, see:

Google Now Censors The Pirate Bay, isoHunt, 4Shared and More
http://torrentfreak.com/google-now-censors-the-pirate-bay-isohunt-4shared-and-more-111123/

Google Censorship Initiative Thwarted by ‘Gee! No Evil!’ Add-On
http://torrentfreak.com/google-censorship-gee-no-evil-add-on-110613/

These developments do not alter the main contention of this article focusing on Google's newly escalated censoring and blacklisting program.

Arbiter of quality

The murky and arbitrary nature of the blacklist is suggested in statements by company executives. For example, as Real News Reporter notes, Google’s Matt Cutts, writing on his Google Plus account, argued that the company has the right to zap sites if they “see a very large fraction of sites on a specific freehost be spammy or low-quality.”

Spammy or low-quality? What does that mean? And that seems to imply an awfully broad and arbitrary brush. What if you were denied service at a restaurant because they refused to serve "gunslingers and people of low quality"? Do you figure you're in the latter category? Moreover, they won't tell you what "quality" rule you violate, nor why. And, oh yeah ... this is a restaurant in Death Valley, the only one with food or water for 100 miles, so you don't really have a viable alternative.

Google's behavior has begun to prompt legal counterattacks, at least in Europe. As described in an article last summer in Search Engine Land (28 June 2011), several companies (including Foundem and Microsoft-owned Ciao) filed formal antitrust complaints against Google with the European Commission in 2010. "Those complaints led directly to the EC’s decision to pursue an antitrust action against Google, which is now in process" reports the website.

Alternatives to Google's power?

Legal fights aside, for most users, Google's blacklist — and Panda — have produced nothing short of a crisis. Google now dominates the online search market — 70 to 80 percent of searches in the USA rely on Google, and an even higher percentage in Europe — 90 percent. Google has reshaped the business world, since businesses now depend on their Internet presence, and in turn rely on Google to help prospective customers find them.

In effect, Google has become a huge pipe, or conduit, through which almost all business marketing, and the vast majority of all informational searching, must be channeled. But now, with its blacklisting of entire sites, and mysterious efforts at manipulation of content via Panda, Google has started down a slippery slope into censoring and asserting control over what can be found on the Web.

Do consumers have any options? Well, there are alternative search engines — Microsoft's Bing.com, for example, which seems to produce even more comprehensive SERPs without a blacklist (at least currently). There's also the possibility of government regulation (lots of luck with that).

The immediate prospect here doesn't look nice. What it could mean is that, if you're searching for effective therapies or cures for that rare disease afflicting your relative, and the exact solution you need happens to be described on a website bumped to page 500 in the search results, or — even worse — placed on the blacklist, because it fails to meet Google's "quality" standards ... Google might expect you to start searching for a funeral parlor, instead. Maybe one that meets its "quality" standards?


2011/12/12

Comments

    0 of 8192 characters used
    Post Comment

    • grand old lady profile image

      Mona Sabalones Gonzalez 3 years ago from Philippines

      Very helpful. I guess it's time to use other search engines aside from Google when I'm doing research on topics with limited information.

    • RTalloni profile image

      RTalloni 5 years ago from the short journey

      Interesting read--thanks!

    • LHwritings profile image
      Author

      Lyndon Henry 5 years ago from Central Texas

      Thanks to all for the interesting and informative feedback. Regarding the information on search results for some of the blacklisted sites mentioned, I've added a section discussing this.

      LH

    • profile image

      The Writers Dog 5 years ago

      Great Hub. Thank you for this information. Mind you, it is not only Google that have gone to the brink of obsessivness when it comes to duplicate content.

    • profile image

      Ghost32 5 years ago

      Prison Planet and Infowars also pull up millions of results on Google (as of 12/13/2011, anyway).

    • Vegefarm profile image

      Vegefarm 5 years ago

      Great Hub,

      I know a close friend who was also affected by Google's evil scheme.Not blacklisted or moved, but by the Google Adsense division. Banded his site from ever participating and could simply not justify their actions. After numerous complaints and such sent, no response was given and he is still sitting in the predicament.

      Google has turned "evil".

      One small thing though, pirate bay, isohunt and 4shared are back up- 1st rankings.

    • Evylyn Rose profile image

      Evylyn Rose 5 years ago from Colorado, USA

      Great hub! Personally, I'm getting tired of spending precious time going through an average of 5-10 search pages just to find what I'm looking for. A lot of the changes to Google (not just Panda) have been annoying to say the least. They definitely aren't the awesome search engine this nerd helped promote through word-of-mouth before "Google" became a household name. I'm sure someone will come along who will create a search engine much closer to what Google used to be, but good luck finding it when they do.