ArtsAutosBooksBusinessEducationEntertainmentFamilyFashionFoodGamesGenderHealthHolidaysHomeHubPagesPersonal FinancePetsPoliticsReligionSportsTechnologyTravel

How Google Index Is Updated

Updated on July 7, 2010

Search Engine Results Index Updates

Whether it is a webpage ranked number one in Google search results or a tiny little junk webpage languishing in the dark corner of the Internet on a test server, the SERPs all come from one place, the index built by programs called spiders.

Search Engine Results Pages, or SERP, are the result of ranking the billions of webpages found all across the web. Each page is found the same way, by tiny automated programs that follow links from here to there and from there to here. They follow links everywhere.

Getting indexed by the search engines will happen eventually, but making it faster requires some basic knowledge of where search engine results come from and how webpages and how hubs get found and indexed by Google and others.

When a user goes to Google.com or Bing.com and runs a search by typing keywords into the search box field, the computers in the Google datacenters run an algorithm against an enormous database of all the webpages that have been indexed. The top search results from that algorithm are reported on the first page of Google search results. The main goal of every search engine company is that the #1 search result on any given search is the "best" or most relevant webpage for the specific search that was run.

In order to achieve this goal, the search engines must first build that database of webpages and all of the content on those websites in order to run their search ranking algorithms. This database of online content is known as the index and it is built by specialized programs called spiders.

How Indexing Search Spiders Work

Spiders are tiny automated bots that crawl across the Internet by following the links they encounter as they index individual webpages. For example, if a search engine spider were crawling the freelance writing business of Arctic Llama homepage, it would (theoretically) follow all of the links from www.arcticllama.com. After following those links, the spider would then read those webpages, add them to the index, and then follow all the links on that page as well.

The idea is that eventually, the search engine spiders would find every webpage and website on the Internet, so long as at least one link somewhere out there points to the page.

In reality, the Internet is actually too big to be indexed in this manner in a useful way. Spiders following links from wherever they start all the way through until they find no more links would take decades to do there jobs and eventually overwhelm not only the servers they were crawling on, but the ones that the search engines have in the giant datacenters as well.

Instead, spiders like the Google search-bots are programmed with limited intelligence and autonomy. They do go out and follow links while cataloging all the content and text they find on each webpage, but they do not continue down the linking superhighway forever. Instead, they don't follow some links and they only follow other links to one level.

That means that in order to get a webpage indexed, it has to get in the way of a running spider.

Spiders crawl frequently updated pages more often, so updates help get pages indexed. They also are more likely to crawl pages with more incoming links, partly due to purely mathematical reasons (the more links, the more likely one is to be followed at random) and also due to webpages with more incoming links being seen as more important or authoritative.

When a spider has run its pre-determined course it terminates, allowing new spiders to be sent out to investigate what the original spider sent back during its run. The process repeats into infinity, indexing the entire Internet along the way, if only for a few seconds each.

Comments

    0 of 8192 characters used
    Post Comment

    • Doc Wordinger profile image

      Doc Wordinger 

      7 years ago from Manchester, UK

      Thanks Llama. This was really interesting and well written. The workings of Google are starting to make sense to me. A bit. I think.

      I look forward to reading some more of your Hubs.

    • Hub Llama profile imageAUTHOR

      Hub Llama 

      8 years ago from Denver, CO

      Mark,

      Glad you liked it.

    • Mark Bradshaw profile image

      Mark Bradshaw 

      8 years ago from England

      very interesting reading thanks

    working

    This website uses cookies

    As a user in the EEA, your approval is needed on a few things. To provide a better website experience, hubpages.com uses cookies (and other similar technologies) and may collect, process, and share personal data. Please choose which areas of our service you consent to our doing so.

    For more information on managing or withdrawing consents and how we handle data, visit our Privacy Policy at: https://hubpages.com/privacy-policy#gdpr

    Show Details
    Necessary
    HubPages Device IDThis is used to identify particular browsers or devices when the access the service, and is used for security reasons.
    LoginThis is necessary to sign in to the HubPages Service.
    Google RecaptchaThis is used to prevent bots and spam. (Privacy Policy)
    AkismetThis is used to detect comment spam. (Privacy Policy)
    HubPages Google AnalyticsThis is used to provide data on traffic to our website, all personally identifyable data is anonymized. (Privacy Policy)
    HubPages Traffic PixelThis is used to collect data on traffic to articles and other pages on our site. Unless you are signed in to a HubPages account, all personally identifiable information is anonymized.
    Amazon Web ServicesThis is a cloud services platform that we used to host our service. (Privacy Policy)
    CloudflareThis is a cloud CDN service that we use to efficiently deliver files required for our service to operate such as javascript, cascading style sheets, images, and videos. (Privacy Policy)
    Google Hosted LibrariesJavascript software libraries such as jQuery are loaded at endpoints on the googleapis.com or gstatic.com domains, for performance and efficiency reasons. (Privacy Policy)
    Features
    Google Custom SearchThis is feature allows you to search the site. (Privacy Policy)
    Google MapsSome articles have Google Maps embedded in them. (Privacy Policy)
    Google ChartsThis is used to display charts and graphs on articles and the author center. (Privacy Policy)
    Google AdSense Host APIThis service allows you to sign up for or associate a Google AdSense account with HubPages, so that you can earn money from ads on your articles. No data is shared unless you engage with this feature. (Privacy Policy)
    Google YouTubeSome articles have YouTube videos embedded in them. (Privacy Policy)
    VimeoSome articles have Vimeo videos embedded in them. (Privacy Policy)
    PaypalThis is used for a registered author who enrolls in the HubPages Earnings program and requests to be paid via PayPal. No data is shared with Paypal unless you engage with this feature. (Privacy Policy)
    Facebook LoginYou can use this to streamline signing up for, or signing in to your Hubpages account. No data is shared with Facebook unless you engage with this feature. (Privacy Policy)
    MavenThis supports the Maven widget and search functionality. (Privacy Policy)
    Marketing
    Google AdSenseThis is an ad network. (Privacy Policy)
    Google DoubleClickGoogle provides ad serving technology and runs an ad network. (Privacy Policy)
    Index ExchangeThis is an ad network. (Privacy Policy)
    SovrnThis is an ad network. (Privacy Policy)
    Facebook AdsThis is an ad network. (Privacy Policy)
    Amazon Unified Ad MarketplaceThis is an ad network. (Privacy Policy)
    AppNexusThis is an ad network. (Privacy Policy)
    OpenxThis is an ad network. (Privacy Policy)
    Rubicon ProjectThis is an ad network. (Privacy Policy)
    TripleLiftThis is an ad network. (Privacy Policy)
    Say MediaWe partner with Say Media to deliver ad campaigns on our sites. (Privacy Policy)
    Remarketing PixelsWe may use remarketing pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to advertise the HubPages Service to people that have visited our sites.
    Conversion Tracking PixelsWe may use conversion tracking pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to identify when an advertisement has successfully resulted in the desired action, such as signing up for the HubPages Service or publishing an article on the HubPages Service.
    Statistics
    Author Google AnalyticsThis is used to provide traffic data and reports to the authors of articles on the HubPages Service. (Privacy Policy)
    ComscoreComScore is a media measurement and analytics company providing marketing data and analytics to enterprises, media and advertising agencies, and publishers. Non-consent will result in ComScore only processing obfuscated personal data. (Privacy Policy)
    Amazon Tracking PixelSome articles display amazon products as part of the Amazon Affiliate program, this pixel provides traffic statistics for those products (Privacy Policy)