How to Tell When Your Hubs Have Been Crawled

1. 87

   Simone Smithposted 11 years ago

   We have added a feature to the Search tab under the stats box on each Hub that enables you to see when that Hub that has last been crawled by search engines.
   
   When an Idle Hub is crawled, search engines will see that it has a noindex tag, which indicates to search engines that the page should not appear in search results. When a Hub falls into Idle status, it may take some time before the Hub is crawled, so it may remain in search engine results for some time. If you update an Idle Hub quickly, right when you see it is no longer Featured, you may bring it out of Idle status before it is crawled, meaning you can avoid having that Hub removed from search engines' indexes entirely.
   
   Do keep in mind, though, that even after a Hub has been crawled, it might take a search engine some time to take it out of the index or add it back in. Essentially, once a Hub has been crawled and a search engine knows what we say it should do with a Hub, the ball is in the search engine's court.
   
   On the flip side, it may take a while between the point in time at which an Idle Hub becomes Featured again and the point at which it is added back to search engine indexes, as the Hub must be crawled first.
   
   We added this feature to make it easier to see why your Hub is being indexed (or not indexed). We hope you find it useful!
   
   For more information on Featured, Pending, and Idle status, stop by our official Learning Center entry.

   report
   1. 0

      RTalloniposted 11 years agoin reply to this

      On more than one level the "why" is huge--thanks! 

      report
   2. 90

      meloncauliposted 11 years agoin reply to this

      What a brilliant idea! This is great Simone. Thanks for letting us know

      report
   3. 82

      K9keystrokesposted 11 years agoin reply to this

      What a fantastic feature! Thank you Simone and the HubPages Team!

      report
2. 92

   ktrappposted 11 years ago

   Very handy to have the last crawl info. here. Thanks.

   report
3. 73

   LetitiaFTposted 11 years ago

   Nifty helpful feature! Thanks!

   report
4. 94

   Heather Saysposted 11 years ago

   Yay!  Thanks for the update and info

   report
5. 82

   tillsontitanposted 11 years ago

   Not only a great idea, but a great help.  This is a benefit not only for idle hubs but hubs we are trying to promote!  Knowing when (and if) Google or Bing have hits on our hubs is just another great weapon in our arsenal of hub promotions.  Thank you, thank you!!!

   report
6. 87

   Simone Smithposted 11 years ago

   Hey everyone! I am so pleased that you find this feature to be helpful.
   
   Many thanks to Paul Deeds and Derek Gulbranson for getting this in order and making sure these things are as clear as possible!

   report
7. 87

   IzzyMposted 11 years ago

   This feature has allowed me to see that my latest hub - http://izzym.hubpages.com/hub/People-Wh … re-Pathway - which covers a very important and potentially life-saving topic, was crawled by the googlebot 8 hours ago, when the noindex tag was still on it.
   
   Now it could be weeks before it comes around again, during which time this hub will not be read except by hubbers and social traffic.
   
   Thanks a lot, Hubpages!

   report
   1. 87

      Simone Smithposted 11 years agoin reply to this

      It may actually take Google some time to remove your Hub from the index, so you might still be getting search traffic for quite some time.

      report
      1. 87

         IzzyMposted 11 years agoin reply to this

         You misunderstood me Simone.
         
         This is a new hub that was put into pending mode.
         
         It was visited by the googlebot during this period, presumably because other hubbers linked it to Facebook and Twitter.
         
         Google saw the noindex tag. It'll likely be weeks before it gets indexed now.
         
         If you are going to hide newly published hubs, then take away all linking from it to social networks to stop the Googlebot from seeing it.

         report
         1. 54

            janderson99posted 11 years agoin reply to this

            The last two hubs I published were visited by G bot within 30 minutes, only to be slapped and told to go away!  Hiding doesn't work when the hubs are sent to followers, etc. I used to love the way new hubs were indexed so quickly, now this good feature works against us and imposes an indexing delay and lost income. Why is this beneficial to established and trusted writers? I have never had a pending hub fail to get the Prince's Kiss and awaken. This is just a pain.

            report
8. 54

   janderson99posted 11 years ago

   This "exclusive" page
   http://janderson99.hubpages.com/hub/Slo … oes-Recipe
   Is listed as being crawled 7 days ago, with an update to say 22 hours ago.
   Yet it does not appear in the SERPS using the search term " Slow Roasted Dried Tomatoes Recipe hubpages"  and cache:http://janderson99.hubpages.com/hub/Slow-Roasted-Dried-Tomatoes-Recipe produces a 404 error
   Any clues???

   report
   1. 54

      janderson99posted 11 years agoin reply to this

      This test works
      
      cache:http://janderson99.hubpages.com/hub/Best-Hair-Color-for-Blue-Eyes-Light-Brown-Green-and-Hazel-Eyes
      
      so what is going on with the tomatoes one!!!

      report
   2. 87

      Simone Smithposted 11 years agoin reply to this

      All we can do is make sure that Hubs are properly tagged and let you know when they are crawled- once they are, it is up to a particular search engine to make a move, I'm afraid!

      report
9. 54

   janderson99posted 11 years ago

   I looked at one of my hubs still in the pending period
   The tool says it was crawled 14 hours ago - with the NOINDEX tag in place.
   This means that the bot won't be back for days or weeks
   This is bad for authors and bad for HP
   The only way I can get pages indexed is to submit them to webmastercentral
   Surely there must be a better way. This is extremely annoying!
   Sending pages to the sin bin tells Google not to index them
   Does HP have any stats on the proportion of pages that are crawled during the pending period and  don't get indexed for days or weeks!
   
   I suspect that EVERY new page gets crawled while pending - the bot sees the NOINDEX tag and goes away - not to return for days or weeks - surely this is bad and should be addressed. The hubber and HP loses income, which can never be regained!!!

   report
   1. 87

      Simone Smithposted 11 years agoin reply to this

      To be fair, most Hubs do not start making significant income until they're quite mature (e.g. have been around for months- if not longer).

      report
      1. 54

         janderson99posted 11 years agoin reply to this

         In my experience that notion is out of date.
         Why doesn't HP introduce 'Trusted Authors' who wouldn't have all their new hubs sent to the 'sin bin'. Many other sites do it. It would free-up scarce resources. A Win Win!
         
         

         report
         1. 0

            EmpressFelicityposted 11 years agoin reply to this

            In theory that's a good idea but I can see problems - it would lay HP open to accusations of favouritism.

            report
            1. 69

               Pearldiverposted 11 years agoin reply to this

               Actually..... That would be a moot point... given the current favoritism practices! 

               report
               1. 87

                  IzzyMposted 11 years agoin reply to this

                  Exactly.
                  
                  We have list of hubbers NOW whose work is featured immediately.
                  
                  Want their names?
                  
                  Most of us have to wait.

                  report
      2. 89

         JayeWisdomposted 11 years agoin reply to this

         Simone....I'm still trying to discover why my traffic hasn't picked back up on my hub (a DIY article) that was, prior to late September, garnering half of my total traffic.  I've lost more than half of all hub traffic and half my earnings this month, even after tweaking my "best" hub.  It still shows the same Google ranking, so I'm bewildered that its traffic is so decreased.  It doesn't help that I'm "technically challenged."  Do you have any explanation about the fate of this hub?  Thanks in advance for any help you can give me.  JAYE

         report
      3. 85

         DzyMsLizzyposted 11 years agoin reply to this

         And just how are they supposed to 'mature' if they get hidden in pending status from the get-go?  Saying 'it's up to the search engine to re-crawl,' is a cop-out, because it was HP who installed the 'no index' PENALTY!!  (For that is what it truly is!)  If it's not indexed, the crawler may not come back for weeks, months or at all...lost income ALL AROUND.  This was NOT a good move on HP's part!
         Being able to "see" when it was last crawled does nothing to get rid of a 'no-index' tag.  We need information on HOW to do THAT!
         Re-write the article?  Add/change tags or keywords?  Or, once it's de-indexed, that's that--too bad, so sad?

         report
      4. 54

         janderson99posted 11 years agoin reply to this

         "To be fair, most Hubs do not start making significant income until they're quite mature (e.g. have been around for months- if not longer)."
         
         Its still lost income - If you impose delays so that an income of $100  that would have occurred in 4 weeks is delayed a week, after 5 weeks your net income is $100 not $200!

         report
10. 54

    janderson99posted 11 years ago

    The tool should test whether the page is ACTUALLY indexed - that is what is important!!!!

    report
    1. 69

       Pearldiverposted 11 years agoin reply to this

       Not to a Smoke and Mirrors Salesperson! 
       
       Maybe the tool can also now upgrade all that missing information on the measuring of hub Viewing Durations?  It seems that half of mine, even though some of the hubs are over two years old haven't yet been measured yet for some rather obscure reason?

       report
    2. 85

       DzyMsLizzyposted 11 years agoin reply to this

       Agreed, janderson99!

       report
11. 83

    David Warrenposted 11 years ago

    Thanks for the info. Great feature that likely as not I wouldn't have found without seeing this.

    report
12. 68

    Rosie2010posted 11 years ago

    Awesome added feature!  Thanks Hubpages!  I never learned how to find out when the bots come a-crawlin' my hubs, so this added feature is very helpful to me.  Cheers! 

    report
13. 82

    pstraubie48posted 11 years ago

    This is one of the best features yet. Thank you so much for giving us insight into it as well.

    report
14. 61

    missoliveposted 11 years ago

    Great job HP! Thanks for adding this feature. I can use all the data I can get.

    report
15. 69

    Beth100posted 11 years ago

    This is an excellent feature!  I have been away for several months, and coming back, I see many useful tools that the HP team has implemented.  It's a great, and wonderful surprise, to come back to a site that has so many useful tools for the writer.
    
    Thank you and kudos to the HP team!

    report
16. 84

    aa liteposted 11 years ago

    HubPages suddenly experiences a significant surge in traffic as all hubbers go through their accounts checking when their hubs were last crawled

    report
    1. 87

       Simone Smithposted 11 years agoin reply to this

       Hahaa!

       report
    2. 89

       JayeWisdomposted 11 years agoin reply to this

       The points DzyMzLzzy and janderson99 made in posts (above) are correct--seeing when hubs are crawled is not helpful if hubbers cannot do anything about indexing. Those are the "tools" HP needs to provide--so we can see if our hubs are indexed and, if not, show us how to ensure they are indexed by Google and other search engines.

       report
17. 93

    viryaboposted 11 years ago

    Its a wonderful feature Simone and HP team. i like that i can know when my site has been crawled.
    
    It will be nice to know when the bots are about to come crawling too 

    report
18. 40

    afmolinajrposted 11 years ago

    this is such a very good feature. now i can manage my hubs even better. Thanks for this one!

    report
19. 71

    rfmoranposted 11 years ago

    Excellent idea for hub management. We can never have enough analytical ideas.

    report
20. 82

    mary615posted 11 years ago

    Many thanks for this feature, HP!  I was pleasantly surprised when I first saw the message:    "congratulations , etc.etc."

    report
21. 77

    twentyfiveposted 11 years ago

    Awesome! Thank you for another great feature Hubpages

    report
22. 92

    HoneyBBposted 11 years ago

    Thanks for sharing this info. I will definitely watch carefully for my hubs to come out of idle status.

    report
23. 89

    Sharkye11posted 11 years ago

    This is a really cool feature. It definitely takes the guess work out of the equation for the authors. Which gives us more time to research and write hubs!

    report
24. 68

    Anamika Sposted 11 years ago

    Cool New Feature... Thanks to HubPages Team

    report
25. 87

    Simone Smithposted 11 years ago

    Thanks for your kind words, feedback, and comments everyone!

    report
26. 0

    EmpressFelicityposted 11 years ago

    Useful feature. Nice one.

    report
27. 68

    sereseusposted 11 years ago

    Thanks for letting us know this.

    report
28. 89

    JayeWisdomposted 11 years ago

    By the way, Simone, I used the new feature to see when my "best" hub was last crawled by search engines.  Not good news. Even though my title uses words from engine searches, Google last crawled the article THREE DAYS AGO, and Bing, two days ago!

    report
29. 90

    JKennyposted 11 years ago

    Great idea! Thanks you Simone and the rest of the HP team. Thank you very much for your efforts and keeping us informed.

    report
30. 68

    carter06posted 11 years ago

    Thanks a lot for Simone it's really helpful for us to be able to see this...appreciate your effort as it gives a broader context to our understanding of our idle hubs...cheers

    report
31. 80

    LiamBeanposted 11 years ago

    This is a really nice feature. As soon as I read this, saw it in slow hub, I used it and made some modifications to existing.
    
    Thanks for the info Simone.

    report