How does google keep track of hubs or any website it indexed?

Jump to Last Post 1-4 of 4 discussions (17 posts)
  1. jackclee lm profile image82
    jackclee lmposted 5 years ago

    Here is a question for experts. I am curious how a hub is tracked over time?
    In specific, suppose I create a new hub or article and it is featured after published on HubPages. After a few days, when the google bot finds it and index it, it appears on google search with some page ranking. After a while, suppose I don’t receive much traffic and Hubpages changed my hub status to non-featured.  Does Google then revise its index of my article or remove it? Or does it stay in the indexed category but just gets pushed down in page ranking?

    The question is really about how the google index is structured. Is it constantly updated with new sites and deleting obsolete sites? Or is it recreated from scratch each time it makes a complete trip around the web?

    My own limited experience seems to point to the first scenario. My hubs are found via google search only when they are featured. Once, it is moved to non-featured status, somehow, it knows to delist it from its index, even though the url is active... how does google know when a hub status has changed?

    1. Susana S profile image93
      Susana Sposted 5 years agoin reply to this

      Google constantly revises its database.

      Most websites have a sitemap which is a file that updates automatically and is sent to google.
      The sitemap gives google information about the pages on the site and if they've been removed or updated etc.

  2. makingamark profile image68
    makingamarkposted 5 years ago

    If any of us knew the answer to that we wouldn't be writing in this forum.

    Of course Google needs to update constantly for all the changes within sites and additions and deletions to existing sites. I've posted blog posts in the past and had them ranking #1 in Google within minutes (and yes I know how to avoid the issue of bias re me looking)

    The one thing I know about Google is that to maintain rankings in Google you need your site to remain fresh, retain traffic and grow links.

    If HubPages delists your hub it's a change and will get picked up by Google the next time the bots do their rounds - however how often a bot looks depends rather on how much traffic you have generated in the recent past.

    1. jackclee lm profile image82
      jackclee lmposted 5 years agoin reply to this

      Thanks for your insight. Someone out there must know how it is not rocket science. How an index is generated based on links and traffic is proprietary algorithm and the speed at which the data is retrieved. The database and its structure is not that hard. There are only a few options for the google implementors...there are bssically two choices, either you create a master list and just add to it and delete from it periodically to keep it current and sort the list based on the proprietary algorithm, or you generate a new listing each time the bots does its crawl...
      There must be a separate list that separates the “featured” from the non-featured and spam sites...

      However, my experience is that some smart spammers have been able to bypass this and get to the top of google page rank regardless, even after reports by users like me, they are still there for whatever reason...perhaps better SEO...

      1. makingamark profile image68
        makingamarkposted 5 years agoin reply to this

        BUT THE WHOLE POINT is that Google do NOT want people to know how it works precisely because them people would game the system!

        Anyway what does it matter?

        Far better to spend time and effort on building better content than trying to second guess Google since by the time you've worked it out they will have changed it again!

        Or are you saying you want to be like the spammers and game the system?

        1. jackclee lm profile image82
          jackclee lmposted 5 years agoin reply to this

          No, I really don’t care either way. It is the technologist in me that would like to know what I am dealing with...
          It seems to be illusive and hard to nail down.
          I like how it use to work when many of hubpages seem to get top billing...
          I wrote about this and was asking how Squidoo, and Hubpages was able to do this?
          I welcome the results then and now it seems to be changing...
          These niche sites are a new foray by HubPages but I am not sure it is the way to go...

          Wikipedia did not do that and they are doing fine...

          1. lobobrandon profile image89
            lobobrandonposted 5 years agoin reply to this

            Wikipedia is doing fine because they never had spam. HP was full of spam. Wikipdia also has links to it from almost all websites worth mentioning. So it is an authority and authority websites are not affected because they have no reason to be affected by Google updates.

            As long as you write good content and know how to structure it so that Google understands your work, you are going to rank well. Provided the domain your content is on has some authority in the eyes of Google. New websites without any links can rank too, more of this has been happening after the March 12th update, which is a good thing.

            1. jackclee lm profile image82
              jackclee lmposted 5 years agoin reply to this

              If that is the case, and I have no reason to doubt you, then the solution should be obvious. Why doesn’t Hubpages get rid of all Spam elements...
              Some of the random ads that appear in our hubs are bordering on spam. They seem to have little to do with the content of my article.
              Hubpages need to establish itself as a quality site and the QAP was suppose to do just that. Perhaps, the answer is to automatically remove all articles that failed the QAP? Why have unfeatured articles here on hubpages if the test of “featured” is the quality?

              1. lobobrandon profile image89
                lobobrandonposted 5 years agoin reply to this

                Again, you cannot compare Hubpages to Wikipedia.

                First of all, recovering from a penalty is difficult for a website. So it does make sense to use new domains. Wikipedia has the authority because they have pages on almost every topic, HP does not have that. Therefore, they build authority by niching down.

                Also, HP does not have the same links as Wikipedia.

                So, just cleaning HP of spam (which they did do by not-featuring spammy articles) helps, but does not give us the traffic, the niche sites can.

                1. jackclee lm profile image82
                  jackclee lmposted 5 years agoin reply to this

                  In my limited experience with the niche sites, I am not seeing much extra traffic for my articles.
                  I have 25 articles on niche sites. That is only 5% of my total articles.
                  If anything, many of my old articles are doing well in google rank where as the niche articles, some of them not so well...
                  At least for me, I don’t see the positive gains from niche sites.
                  I am keeping an open mind and wait and see how it goes.

                  1. lobobrandon profile image89
                    lobobrandonposted 5 years agoin reply to this

                    If you are not seeing traffic after moving to a niche site it is most likely because no one is searching for what you wrote. A change in title may help.

  3. Susana S profile image93
    Susana Sposted 5 years ago

    Have a read up on Google rankbrain, it's a core part of the algorithm.

    1. jackclee lm profile image82
      jackclee lmposted 5 years agoin reply to this

      Thanks for the link, very interesting and informative.

    2. tsmog profile image80
      tsmogposted 5 years agoin reply to this

      Thanks for the recommendation to this article. It brought a clearer understanding to the keyword usage, titles, and the importance of dwell time.

  4. lobobrandon profile image89
    lobobrandonposted 5 years ago

    To be honest MAM, Google does make this knowledge public. Jacklee is not asking us how the ranking algorithm works, but how Googles database works.

    Google gets new pages into its database like this:

    The page is linked to it from a pre-existing page in the database that google crawled recently.
    The page is added to the sitemap that you submit to google via Google webmaster tools
    The page is fetched and rendered via Google webmaster tools
    The page is added to google by typing in submit URL to google on google search
    The page is linked to from a page on its own site and when google crawls your site again it finds this page.

    Once a page is indexed Google assigns it some weighting factor based on the website it belongs to and Google recrawls this occasionally. Can be every few seconds for most news sites or after months if you have a one or two-page website.

    When your hub is unfeatured it gets a no-index tag. This tells google that you do not want the page to show up in search results. Google acknowledges this and removes it from the database once it notices this tag. So when Google re-crawls your still live URL and sees this tag, the page is removed from the database.

    1. makingamark profile image68
      makingamarkposted 5 years agoin reply to this

      Point taken - I was assuming he meant HOW the algorithm works once pages are recognised

      Are you assuming that everyone knows about the Google Search Console and how to add a site map? (i.e. "webmaster tools" went on its way some time ago)

      I think not!

      1. lobobrandon profile image89
        lobobrandonposted 5 years agoin reply to this

        I'm not assuming anything, the question was about how the index works, so I stated the different ways you can add a page to this database.


This website uses cookies

As a user in the EEA, your approval is needed on a few things. To provide a better website experience, uses cookies (and other similar technologies) and may collect, process, and share personal data. Please choose which areas of our service you consent to our doing so.

For more information on managing or withdrawing consents and how we handle data, visit our Privacy Policy at:

Show Details
HubPages Device IDThis is used to identify particular browsers or devices when the access the service, and is used for security reasons.
LoginThis is necessary to sign in to the HubPages Service.
Google RecaptchaThis is used to prevent bots and spam. (Privacy Policy)
AkismetThis is used to detect comment spam. (Privacy Policy)
HubPages Google AnalyticsThis is used to provide data on traffic to our website, all personally identifyable data is anonymized. (Privacy Policy)
HubPages Traffic PixelThis is used to collect data on traffic to articles and other pages on our site. Unless you are signed in to a HubPages account, all personally identifiable information is anonymized.
Amazon Web ServicesThis is a cloud services platform that we used to host our service. (Privacy Policy)
CloudflareThis is a cloud CDN service that we use to efficiently deliver files required for our service to operate such as javascript, cascading style sheets, images, and videos. (Privacy Policy)
Google Hosted LibrariesJavascript software libraries such as jQuery are loaded at endpoints on the or domains, for performance and efficiency reasons. (Privacy Policy)
Google Custom SearchThis is feature allows you to search the site. (Privacy Policy)
Google MapsSome articles have Google Maps embedded in them. (Privacy Policy)
Google ChartsThis is used to display charts and graphs on articles and the author center. (Privacy Policy)
Google AdSense Host APIThis service allows you to sign up for or associate a Google AdSense account with HubPages, so that you can earn money from ads on your articles. No data is shared unless you engage with this feature. (Privacy Policy)
Google YouTubeSome articles have YouTube videos embedded in them. (Privacy Policy)
VimeoSome articles have Vimeo videos embedded in them. (Privacy Policy)
PaypalThis is used for a registered author who enrolls in the HubPages Earnings program and requests to be paid via PayPal. No data is shared with Paypal unless you engage with this feature. (Privacy Policy)
Facebook LoginYou can use this to streamline signing up for, or signing in to your Hubpages account. No data is shared with Facebook unless you engage with this feature. (Privacy Policy)
MavenThis supports the Maven widget and search functionality. (Privacy Policy)
Google AdSenseThis is an ad network. (Privacy Policy)
Google DoubleClickGoogle provides ad serving technology and runs an ad network. (Privacy Policy)
Index ExchangeThis is an ad network. (Privacy Policy)
SovrnThis is an ad network. (Privacy Policy)
Facebook AdsThis is an ad network. (Privacy Policy)
Amazon Unified Ad MarketplaceThis is an ad network. (Privacy Policy)
AppNexusThis is an ad network. (Privacy Policy)
OpenxThis is an ad network. (Privacy Policy)
Rubicon ProjectThis is an ad network. (Privacy Policy)
TripleLiftThis is an ad network. (Privacy Policy)
Say MediaWe partner with Say Media to deliver ad campaigns on our sites. (Privacy Policy)
Remarketing PixelsWe may use remarketing pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to advertise the HubPages Service to people that have visited our sites.
Conversion Tracking PixelsWe may use conversion tracking pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to identify when an advertisement has successfully resulted in the desired action, such as signing up for the HubPages Service or publishing an article on the HubPages Service.
Author Google AnalyticsThis is used to provide traffic data and reports to the authors of articles on the HubPages Service. (Privacy Policy)
ComscoreComScore is a media measurement and analytics company providing marketing data and analytics to enterprises, media and advertising agencies, and publishers. Non-consent will result in ComScore only processing obfuscated personal data. (Privacy Policy)
Amazon Tracking PixelSome articles display amazon products as part of the Amazon Affiliate program, this pixel provides traffic statistics for those products (Privacy Policy)
ClickscoThis is a data management platform studying reader behavior (Privacy Policy)