How does google keep track of hubs or any website it indexed?

1. 76

   jackclee lmposted 7 years ago

   Here is a question for experts. I am curious how a hub is tracked over time?
   In specific, suppose I create a new hub or article and it is featured after published on HubPages. After a few days, when the google bot finds it and index it, it appears on google search with some page ranking. After a while, suppose I don’t receive much traffic and Hubpages changed my hub status to non-featured.  Does Google then revise its index of my article or remove it? Or does it stay in the indexed category but just gets pushed down in page ranking?
   
   The question is really about how the google index is structured. Is it constantly updated with new sites and deleting obsolete sites? Or is it recreated from scratch each time it makes a complete trip around the web?
   
   My own limited experience seems to point to the first scenario. My hubs are found via google search only when they are featured. Once, it is moved to non-featured status, somehow, it knows to delist it from its index, even though the url is active... how does google know when a hub status has changed?

   replyreport
   1. 96

      Susana Sposted 7 years agoin reply to this

      Google constantly revises its database.
      
      Most websites have a sitemap which is a file that updates automatically and is sent to google.
      The sitemap gives google information about the pages on the site and if they've been removed or updated etc.
      
      https://www.ccmarketingonline.com/what-is-a-sitemap/

      replyreport
2. 69

   makingamarkposted 7 years ago

   If any of us knew the answer to that we wouldn't be writing in this forum.
   
   Of course Google needs to update constantly for all the changes within sites and additions and deletions to existing sites. I've posted blog posts in the past and had them ranking #1 in Google within minutes (and yes I know how to avoid the issue of bias re me looking)
   
   The one thing I know about Google is that to maintain rankings in Google you need your site to remain fresh, retain traffic and grow links.
   
   If HubPages delists your hub it's a change and will get picked up by Google the next time the bots do their rounds - however how often a bot looks depends rather on how much traffic you have generated in the recent past.

   replyreport
   1. 76

      jackclee lmposted 7 years agoin reply to this

      Thanks for your insight. Someone out there must know how it works...it is not rocket science. How an index is generated based on links and traffic is proprietary algorithm and the speed at which the data is retrieved. The database and its structure is not that hard. There are only a few options for the google implementors...there are bssically two choices, either you create a master list and just add to it and delete from it periodically to keep it current and sort the list based on the proprietary algorithm, or you generate a new listing each time the bots does its crawl...
      There must be a separate list that separates the “featured” from the non-featured and spam sites...
      
      However, my experience is that some smart spammers have been able to bypass this and get to the top of google page rank regardless, even after reports by users like me, they are still there for whatever reason...perhaps better SEO...

      replyreport
      1. 69

         makingamarkposted 7 years agoin reply to this

         BUT THE WHOLE POINT is that Google do NOT want people to know how it works precisely because them people would game the system!
         
         Anyway what does it matter?
         
         Far better to spend time and effort on building better content than trying to second guess Google since by the time you've worked it out they will have changed it again!
         
         Or are you saying you want to be like the spammers and game the system?

         replyreport
         1. 76

            jackclee lmposted 7 years agoin reply to this

            No, I really don’t care either way. It is the technologist in me that would like to know what I am dealing with...
            It seems to be illusive and hard to nail down.
            I like how it use to work when many of hubpages seem to get top billing...
            I wrote about this and was asking how Squidoo, and Hubpages was able to do this?
            I welcome the results then and now it seems to be changing...
            These niche sites are a new foray by HubPages but I am not sure it is the way to go...
            
            Wikipedia did not do that and they are doing fine...

            replyreport
            1. 77

               lobobrandonposted 7 years agoin reply to this

               Wikipedia is doing fine because they never had spam. HP was full of spam. Wikipdia also has links to it from almost all websites worth mentioning. So it is an authority and authority websites are not affected because they have no reason to be affected by Google updates.
               
               As long as you write good content and know how to structure it so that Google understands your work, you are going to rank well. Provided the domain your content is on has some authority in the eyes of Google. New websites without any links can rank too, more of this has been happening after the March 12th update, which is a good thing.

               replyreport
               1. 76

                  jackclee lmposted 7 years agoin reply to this

                  If that is the case, and I have no reason to doubt you, then the solution should be obvious. Why doesn’t Hubpages get rid of all Spam elements...
                  Some of the random ads that appear in our hubs are bordering on spam. They seem to have little to do with the content of my article.
                  Hubpages need to establish itself as a quality site and the QAP was suppose to do just that. Perhaps, the answer is to automatically remove all articles that failed the QAP? Why have unfeatured articles here on hubpages if the test of “featured” is the quality?

                  replyreport
                  1. 77

                     lobobrandonposted 7 years agoin reply to this

                     Again, you cannot compare Hubpages to Wikipedia.
                     
                     First of all, recovering from a penalty is difficult for a website. So it does make sense to use new domains. Wikipedia has the authority because they have pages on almost every topic, HP does not have that. Therefore, they build authority by niching down.
                     
                     Also, HP does not have the same links as Wikipedia.
                     
                     So, just cleaning HP of spam (which they did do by not-featuring spammy articles) helps, but does not give us the traffic, the niche sites can.

                     replyreport
                     1. 76

                        jackclee lmposted 7 years agoin reply to this

                        In my limited experience with the niche sites, I am not seeing much extra traffic for my articles.
                        I have 25 articles on niche sites. That is only 5% of my total articles.
                        If anything, many of my old articles are doing well in google rank where as the niche articles, some of them not so well...
                        At least for me, I don’t see the positive gains from niche sites.
                        I am keeping an open mind and wait and see how it goes.

                        replyreport
                        1. 77

                           lobobrandonposted 7 years agoin reply to this

                           If you are not seeing traffic after moving to a niche site it is most likely because no one is searching for what you wrote. A change in title may help.

                           replyreport
3. 96

   Susana Sposted 7 years ago

   Have a read up on Google rankbrain, it's a core part of the algorithm. https://backlinko.com/google-rankbrain-seo

   replyreport
   1. 76

      jackclee lmposted 7 years agoin reply to this

      Thanks for the link, very interesting and informative.

      replyreport
   2. 88

      tsmogposted 7 years agoin reply to this

      Thanks for the recommendation to this article. It brought a clearer understanding to the keyword usage, titles, and the importance of dwell time.

      replyreport
4. 77

   lobobrandonposted 7 years ago

   To be honest MAM, Google does make this knowledge public. Jacklee is not asking us how the ranking algorithm works, but how Googles database works.
   
   Google gets new pages into its database like this:
   
   The page is linked to it from a pre-existing page in the database that google crawled recently.
   The page is added to the sitemap that you submit to google via Google webmaster tools
   The page is fetched and rendered via Google webmaster tools
   The page is added to google by typing in submit URL to google on google search
   The page is linked to from a page on its own site and when google crawls your site again it finds this page.
   
   Once a page is indexed Google assigns it some weighting factor based on the website it belongs to and Google recrawls this occasionally. Can be every few seconds for most news sites or after months if you have a one or two-page website.
   
   When your hub is unfeatured it gets a no-index tag. This tells google that you do not want the page to show up in search results. Google acknowledges this and removes it from the database once it notices this tag. So when Google re-crawls your still live URL and sees this tag, the page is removed from the database.

   replyreport
   1. 69

      makingamarkposted 7 years agoin reply to this

      Point taken - I was assuming he meant HOW the algorithm works once pages are recognised
      
      Are you assuming that everyone knows about the Google Search Console and how to add a site map? (i.e. "webmaster tools" went on its way some time ago)
      
      I think not!

      replyreport
      1. 77

         lobobrandonposted 7 years agoin reply to this

         I'm not assuming anything, the question was about how the index works, so I stated the different ways you can add a page to this database.

         replyreport