Hub index and crawl timing

1. 96

   EricDockettposted 9 years ago

   The last few Hubs I published (on a different account) were not indexed by search engines as quickly as my Hubs have been in the past. They were always indexed within a few hours after passing QAP. Now it is taking much longer, and in some cases a day later they still aren't indexed. I have been adding them to search via WMT at this point instead of waiting longer.
   
   Is this possibly a bug? Has there been some change to this site recently that might be slowing down bot visits or preventing them from getting where they need to go?

   replyreport
   1. 31

      Writer Foxposted 9 years agoin reply to this

      As soon as your Hub clears QAP and is featured, submit it yourself using your account at Google Webmaster Tools.
      1. Choose 'Crawl' from the left menu.
      2. Choose 'Fetch as Google' from the left menu.
      3. Submit the URL (note: include /Hub/).
      4. When the URL is fetched, click on the choice 'Submit to Index'.
      5. Then do a Google search for your Hub by putting the first few lines in quotes and it will appear.
      
      You can do the same thing for Bing.  Also, it is a good idea to Ping a new Hub.

      replyreport
      1. 96

         EricDockettposted 9 years agoin reply to this

         I have been submitting them through the "Fetch as Google" utility when I see they haven't been indexed within a reasonable time. It is a good fix and I do it when I have to, but I'd really prefer to have the confidence that my Hub will be crawled and indexed in a timely manner after publication. It's not really practical to sit around staring at my stats and waiting for a new Hub to clear QAP so I can ask search engines to crawl it.
         
         What method do you recommend for pinging a new Hub? That's something I haven't thought about in a long time.

         replyreport
      2. 82

         Shades-of-truthposted 9 years agoin reply to this

         Question for Writer Fox:
         
         What exactly do you mean by #3 - "(note: include /Hub/)"?

         replyreport
         1. 31

            Writer Foxposted 9 years agoin reply to this

            When you go to enter the URL of your new Hub, Google already has the first portion:
            
            
            So, you have to enter: hub/URL-Hub-Title

            replyreport
            1. 82

               Shades-of-truthposted 9 years agoin reply to this

               Thank you!

               replyreport
               1. 31

                  Writer Foxposted 9 years agoin reply to this

                  You're welcome.  Remember, you can't use this feature if your Hub is an EC Hub, because then it is not on your subdomain.

                  replyreport
   2. 71

      Matthew Meyerposted 9 years agoin reply to this

      I shared this information with our engineering team and we looked over the data for your recent Hubs. The time from featured status until a Hub is indexed varies among search engines.
      
      For your specific Hubs there were cases where Yahoo or MSN indexed a Hub in 2 or 3 hours, but it took Google almost 2 days to crawl the same Hub. MSN and Yahoo both crawling content so quickly, points to this not being something on the HubPages end that would be affecting when something is crawled (all crawl bots see the same robots.txt).
      
      It looks like you may be seeing a change in how Google is crawling content and we do not have any direct control over when they choose to crawl/index content.
      This does not appear to be a bug with our site.
      
      You can see the last crawl date on the "search" tab in the author center on Hubs.

      replyreport
      1. 96

         EricDockettposted 9 years agoin reply to this

         Thanks for looking into this Matthew. I do appreciate it.  I guess I'll just have to stay on top of new Hubs and submit them myself if they aren't crawled within a reasonable timeframe.

         replyreport