jump to last post 1-31 of 31 discussions (51 posts)

How to Tell When Your Hubs Have Been Crawled

  1. Simone Smith profile image91
    Simone Smithposted 4 years ago

    We have added a feature to the Search tab under the stats box on each Hub that enables you to see when that Hub that has last been crawled by search engines.
    http://s4.hubimg.com/u/7292927_f520.jpg
    When an Idle Hub is crawled, search engines will see that it has a noindex tag, which indicates to search engines that the page should not appear in search results. When a Hub falls into Idle status, it may take some time before the Hub is crawled, so it may remain in search engine results for some time. If you update an Idle Hub quickly, right when you see it is no longer Featured, you may bring it out of Idle status before it is crawled, meaning you can avoid having that Hub removed from search engines' indexes entirely.

    Do keep in mind, though, that even after a Hub has been crawled, it might take a search engine some time to take it out of the index or add it back in. Essentially, once a Hub has been crawled and a search engine knows what we say it should do with a Hub, the ball is in the search engine's court.

    On the flip side, it may take a while between the point in time at which an Idle Hub becomes Featured again and the point at which it is added back to search engine indexes, as the Hub must be crawled first.

    We added this feature to make it easier to see why your Hub is being indexed (or not indexed). We hope you find it useful!

    For more information on Featured, Pending, and Idle status, stop by our official Learning Center entry.

    1. RTalloni profile image86
      RTalloniposted 4 years ago in reply to this

      On more than one level the "why" is huge--thanks!  smile

    2. meloncauli profile image100
      meloncauliposted 4 years ago in reply to this

      What a brilliant idea! This is great Simone. Thanks for letting us know smile

      1. K9keystrokes profile image92
        K9keystrokesposted 4 years ago in reply to this

        What a fantastic feature! Thank you Simone and the HubPages Team! big_smile

    3. ktrapp profile image91
      ktrappposted 4 years ago

      Very handy to have the last crawl info. here. Thanks.

    4. LetitiaFT profile image82
      LetitiaFTposted 4 years ago

      Nifty helpful feature! Thanks!

    5. Heather Says profile image97
      Heather Saysposted 4 years ago

      Yay!  Thanks for the update and info smile

    6. tillsontitan profile image91
      tillsontitanposted 4 years ago

      Not only a great idea, but a great help.  This is a benefit not only for idle hubs but hubs we are trying to promote!  Knowing when (and if) Google or Bing have hits on our hubs is just another great weapon in our arsenal of hub promotions.  Thank you, thank you!!!

    7. Simone Smith profile image91
      Simone Smithposted 4 years ago

      Hey everyone! I am so pleased that you find this feature to be helpful.

      Many thanks to Paul Deeds and Derek Gulbranson for getting this in order and making sure these things are as clear as possible!

    8. IzzyM profile image86
      IzzyMposted 4 years ago

      This feature has allowed me to see that my latest hub - http://izzym.hubpages.com/hub/People-Wh … re-Pathway - which covers a very important and potentially life-saving topic, was crawled by the googlebot 8 hours ago, when the noindex tag was still on it.

      Now it could be weeks before it comes around again, during which time this hub will not be read except by hubbers and social traffic.

      Thanks a lot, Hubpages!

      1. Simone Smith profile image91
        Simone Smithposted 4 years ago in reply to this

        It may actually take Google some time to remove your Hub from the index, so you might still be getting search traffic for quite some time. smile

        1. IzzyM profile image86
          IzzyMposted 4 years ago in reply to this

          You misunderstood me Simone.

          This is a new hub that was put into pending mode.

          It was visited by the googlebot during this period, presumably because other hubbers linked it to Facebook and Twitter.

          Google saw the noindex tag. It'll likely be weeks before it gets indexed now.

          If you are going to hide newly published hubs, then take away all linking from it to social networks to stop the Googlebot from seeing it.

          1. janderson99 profile image85
            janderson99posted 4 years ago in reply to this

            The last two hubs I published were visited by G bot within 30 minutes, only to be slapped and told to go away!  Hiding doesn't work when the hubs are sent to followers, etc. I used to love the way new hubs were indexed so quickly, now this good feature works against us and imposes an indexing delay and lost income. Why is this beneficial to established and trusted writers? I have never had a pending hub fail to get the Prince's Kiss and awaken. This is just a pain.

    9. janderson99 profile image85
      janderson99posted 4 years ago

      This "exclusive" page
      http://janderson99.hubpages.com/hub/Slo … oes-Recipe
      Is listed as being crawled 7 days ago, with an update to say 22 hours ago.
      Yet it does not appear in the SERPS using the search term " Slow Roasted Dried Tomatoes Recipe hubpages"  and cache:http://janderson99.hubpages.com/hub/Slow-Roasted-Dried-Tomatoes-Recipe produces a 404 error
      Any clues???

      1. janderson99 profile image85
        janderson99posted 4 years ago in reply to this

        This test works

        cache:http://janderson99.hubpages.com/hub/Best-Hair-Color-for-Blue-Eyes-Light-Brown-Green-and-Hazel-Eyes

        so what is going on with the tomatoes one!!!

      2. Simone Smith profile image91
        Simone Smithposted 4 years ago in reply to this

        All we can do is make sure that Hubs are properly tagged and let you know when they are crawled- once they are, it is up to a particular search engine to make a move, I'm afraid!

      3. janderson99 profile image85
        janderson99posted 4 years ago

        I looked at one of my hubs still in the pending period
        The tool says it was crawled 14 hours ago - with the NOINDEX tag in place.
        This means that the bot won't be back for days or weeks
        This is bad for authors and bad for HP
        The only way I can get pages indexed is to submit them to webmastercentral
        Surely there must be a better way. This is extremely annoying!
        Sending pages to the sin bin tells Google not to index them
        Does HP have any stats on the proportion of pages that are crawled during the pending period and  don't get indexed for days or weeks!

        I suspect that EVERY new page gets crawled while pending - the bot sees the NOINDEX tag and goes away - not to return for days or weeks - surely this is bad and should be addressed. The hubber and HP loses income, which can never be regained!!!

        1. Simone Smith profile image91
          Simone Smithposted 4 years ago in reply to this

          To be fair, most Hubs do not start making significant income until they're quite mature (e.g. have been around for months- if not longer). smile

          1. janderson99 profile image85
            janderson99posted 4 years ago in reply to this

            In my experience that notion is out of date.
            Why doesn't HP introduce 'Trusted Authors' who wouldn't have all their new hubs sent to the 'sin bin'. Many other sites do it. It would free-up scarce resources. A Win Win!

            http://s3.hubimg.com/u/7296726_f248.jpg

            1. EmpressFelicity profile image84
              EmpressFelicityposted 4 years ago in reply to this

              In theory that's a good idea but I can see problems - it would lay HP open to accusations of favouritism.

              1. Pearldiver profile image86
                Pearldiverposted 4 years ago in reply to this

                Actually..... That would be a moot point... given the current favoritism practices!  lol

                1. IzzyM profile image86
                  IzzyMposted 4 years ago in reply to this

                  Exactly.

                  We have list of hubbers NOW whose work is featured immediately.

                  Want their names?

                  Most of us have to wait.

          2. JayeWisdom profile image93
            JayeWisdomposted 4 years ago in reply to this

            Simone....I'm still trying to discover why my traffic hasn't picked back up on my hub (a DIY article) that was, prior to late September, garnering half of my total traffic.  I've lost more than half of all hub traffic and half my earnings this month, even after tweaking my "best" hub.  It still shows the same Google ranking, so I'm bewildered that its traffic is so decreased.  It doesn't help that I'm "technically challenged."  Do you have any explanation about the fate of this hub?  Thanks in advance for any help you can give me.  JAYE

          3. DzyMsLizzy profile image91
            DzyMsLizzyposted 4 years ago in reply to this

            And just how are they supposed to 'mature' if they get hidden in pending status from the get-go?  Saying 'it's up to the search engine to re-crawl,' is a cop-out, because it was HP who installed the 'no index' PENALTY!!  (For that is what it truly is!)  If it's not indexed, the crawler may not come back for weeks, months or at all...lost income ALL AROUND.  This was NOT a good move on HP's part!
            Being able to "see" when it was last crawled does nothing to get rid of a 'no-index' tag.  We need information on HOW to do THAT!
            Re-write the article?  Add/change tags or keywords?  Or, once it's de-indexed, that's that--too bad, so sad?

          4. janderson99 profile image85
            janderson99posted 4 years ago in reply to this

            "To be fair, most Hubs do not start making significant income until they're quite mature (e.g. have been around for months- if not longer)."

            Its still lost income - If you impose delays so that an income of $100  that would have occurred in 4 weeks is delayed a week, after 5 weeks your net income is $100 not $200!

      4. janderson99 profile image85
        janderson99posted 4 years ago

        The tool should test whether the page is ACTUALLY indexed - that is what is important!!!!

        1. Pearldiver profile image86
          Pearldiverposted 4 years ago in reply to this

          Not to a Smoke and Mirrors Salesperson!  sad

          Maybe the tool can also now upgrade all that missing information on the measuring of hub Viewing Durations?  It seems that half of mine, even though some of the hubs are over two years old haven't yet been measured yet for some rather obscure reason?

        2. DzyMsLizzy profile image91
          DzyMsLizzyposted 4 years ago in reply to this

          Agreed, janderson99!

      5. David Warren profile image84
        David Warrenposted 4 years ago

        Thanks for the info. Great feature that likely as not I wouldn't have found without seeing this.

      6. Rosie2010 profile image84
        Rosie2010posted 4 years ago

        Awesome added feature!  Thanks Hubpages!  I never learned how to find out when the bots come a-crawlin' my hubs, so this added feature is very helpful to me.  Cheers!  smile

      7. pstraubie48 profile image88
        pstraubie48posted 4 years ago

        This is one of the best features yet. Thank you so much for giving us insight into it as well.

      8. missolive profile image93
        missoliveposted 4 years ago

        Great job HP! Thanks for adding this feature. I can use all the data I can get. smile

      9. Beth100 profile image83
        Beth100posted 4 years ago

        This is an excellent feature!  I have been away for several months, and coming back, I see many useful tools that the HP team has implemented.  It's a great, and wonderful surprise, to come back to a site that has so many useful tools for the writer.

        Thank you and kudos to the HP team!

      10. aa lite profile image89
        aa liteposted 4 years ago

        HubPages suddenly experiences a significant surge in traffic as all hubbers go through their accounts checking when their hubs were last crawled smile

        1. Simone Smith profile image91
          Simone Smithposted 4 years ago in reply to this

          Hahaa!

        2. JayeWisdom profile image93
          JayeWisdomposted 4 years ago in reply to this

          The points DzyMzLzzy and janderson99 made in posts (above) are correct--seeing when hubs are crawled is not helpful if hubbers cannot do anything about indexing. Those are the "tools" HP needs to provide--so we can see if our hubs are indexed and, if not, show us how to ensure they are indexed by Google and other search engines.

      11. viryabo profile image86
        viryaboposted 4 years ago

        Its a wonderful feature Simone and HP team. smile i like that i can know when my site has been crawled.

        It will be nice to know when the bots are about to come crawling too  tongue

      12. afmolinajr profile image79
        afmolinajrposted 4 years ago

        this is such a very good feature. now i can manage my hubs even better. Thanks for this one!

      13. rfmoran profile image91
        rfmoranposted 4 years ago

        Excellent idea for hub management. We can never have enough analytical ideas.

      14. mary615 profile image94
        mary615posted 4 years ago

        Many thanks for this feature, HP!  I was pleasantly surprised when I first saw the message:    "congratulations , etc.etc."

      15. twentyfive profile image78
        twentyfiveposted 4 years ago

        Awesome! Thank you for another great feature Hubpages smile

      16. HoneyBB profile image86
        HoneyBBposted 4 years ago

        Thanks for sharing this info. I will definitely watch carefully for my hubs to come out of idle status.

      17. Sharkye11 profile image94
        Sharkye11posted 4 years ago

        This is a really cool feature. It definitely takes the guess work out of the equation for the authors. Which gives us more time to research and write hubs!

      18. Anamika S profile image69
        Anamika Sposted 4 years ago

        Cool New Feature... Thanks to HubPages Team

      19. Simone Smith profile image91
        Simone Smithposted 4 years ago

        Thanks for your kind words, feedback, and comments everyone!

      20. EmpressFelicity profile image84
        EmpressFelicityposted 4 years ago

        Useful feature. Nice one.

      21. sereseus profile image77
        sereseusposted 4 years ago

        Thanks for letting us know this.

      22. JayeWisdom profile image93
        JayeWisdomposted 4 years ago

        By the way, Simone, I used the new feature to see when my "best" hub was last crawled by search engines.  Not good news. Even though my title uses words from engine searches, Google last crawled the article THREE DAYS AGO, and Bing, two days ago!

      23. JKenny profile image92
        JKennyposted 4 years ago

        Great idea! Thanks you Simone and the rest of the HP team. Thank you very much for your efforts and keeping us informed.

      24. carter06 profile image88
        carter06posted 4 years ago

        Thanks a lot for Simone it's really helpful for us to be able to see this...appreciate your effort as it gives a broader context to our understanding of our idle hubs...cheers

      25. LiamBean profile image88
        LiamBeanposted 4 years ago

        This is a really nice feature. As soon as I read this, saw it in slow hub, I used it and made some modifications to existing.

        Thanks for the info Simone.

      Closed to reply
       
      working