jump to last post 1-1 of 1 discussions (9 posts)

Hub index and crawl timing

  1. EricDockett profile image92
    EricDockettposted 2 years ago

    The last few Hubs I published (on a different account) were not indexed by search engines as quickly as my Hubs have been in the past. They were always indexed within a few hours after passing QAP. Now it is taking much longer, and in some cases a day later they still aren't indexed. I have been adding them to search via WMT at this point instead of waiting longer.

    Is this possibly a bug? Has there been some change to this site recently that might be slowing down bot visits or preventing them from getting where they need to go?

    1. Writer Fox profile image78
      Writer Foxposted 2 years ago in reply to this

      As soon as your Hub clears QAP and is featured, submit it yourself using your account at Google Webmaster Tools.
      1. Choose 'Crawl' from the left menu.
      2. Choose 'Fetch as Google' from the left menu.
      3. Submit the URL (note: include /Hub/).
      4. When the URL is fetched, click on the choice 'Submit to Index'.
      5. Then do a Google search for your Hub by putting the first few lines in quotes and it will appear.

      You can do the same thing for Bing.  Also, it is a good idea to Ping a new Hub.

      1. EricDockett profile image92
        EricDockettposted 2 years ago in reply to this

        I have been submitting them through the "Fetch as Google" utility when I see they haven't been indexed within a reasonable time. It is a good fix and I do it when I have to, but I'd really prefer to have the confidence that my Hub will be crawled and indexed in a timely manner after publication. It's not really practical to sit around staring at my stats and waiting for a new Hub to clear QAP so I can ask search engines to crawl it.

        What method do you recommend for pinging a new Hub? That's something I haven't thought about in a long time.

      2. Shades-of-truth profile image86
        Shades-of-truthposted 2 years ago in reply to this

        Question for Writer Fox:

        What exactly do you mean by #3 - "(note: include /Hub/)"?

        1. Writer Fox profile image78
          Writer Foxposted 2 years ago in reply to this

          When you go to enter the URL of your new Hub, Google already has the first portion:
          http://s1.hubimg.com/u/12329534.png

          So, you have to enter: hub/URL-Hub-Title

          1. Shades-of-truth profile image86
            Shades-of-truthposted 2 years ago in reply to this

            Thank you!

            1. Writer Fox profile image78
              Writer Foxposted 2 years ago in reply to this

              You're welcome.  Remember, you can't use this feature if your Hub is an EC Hub, because then it is not on your subdomain.

    2. Matthew Meyer profile image74
      Matthew Meyerposted 2 years ago in reply to this

      I shared this information with our engineering team and we looked over the data for your recent Hubs. The time from featured status until a Hub is indexed varies among search engines.

      For your specific Hubs there were cases where Yahoo or MSN indexed a Hub in 2 or 3 hours, but it took Google almost 2 days to crawl the same Hub. MSN and Yahoo both crawling content so quickly, points to this not being something on the HubPages end that would be affecting when something is crawled (all crawl bots see the same robots.txt).

      It looks like you may be seeing a change in how Google is crawling content and we do not have any direct control over when they choose to crawl/index content.
      This does not appear to be a bug with our site.

      You can see the last crawl date on the "search" tab in the author center on Hubs.

      1. EricDockett profile image92
        EricDockettposted 2 years ago in reply to this

        Thanks for looking into this Matthew. I do appreciate it.  I guess I'll just have to stay on top of new Hubs and submit them myself if they aren't crawled within a reasonable timeframe.

 
working