jump to last post 1-9 of 9 discussions (31 posts)

Unindexed Hub Listed as Having a "Google Crawl"

  1. agilitymach profile image98
    agilitymachposted 4 years ago

    I have a hub that went unfeatured several days back.  It was my second hub, and it really was in dire need of rewriting.  After doing an extensive revision, I sent it out to the QAP.  Then I checked the stats to see how many views it has gotten to date.

    I was surprised to see that a Google bot had crawled it two hours ago, and a Bing bot had crawled it 12 hours ago.   Aren't unfeatured hubs supposed to be "unseen" by the search engines? 

    Any idea why the bot crawls?  This isn't a huge issue, but I am curious.  The hub is now listed as "pending."

    1. AMFredenburg profile image78
      AMFredenburgposted 4 years ago in reply to this

      How could you tell that the bots were crawling?

    2. Matthew Meyer profile image74
      Matthew Meyerposted 4 years ago in reply to this

      Crawling and being indexed in a search engine are different things.

      1. AMFredenburg profile image78
        AMFredenburgposted 4 years ago in reply to this

        Okay, I am being crawled; how do I find out if I am indexed?

        1. Matthew Meyer profile image74
          Matthew Meyerposted 4 years ago in reply to this

          By searching for your Hub in the Search Engine in which you would like to determine indexed status.

          So if I was trying to see if this Hub is indexed in Google:
          http://hubpages.com/learningcenter/Diag … r-problems

          You would search this exact string in Google:
          "site:hubpages.com/learningcenter/Diagnosing-common-browser-problems/"

          So the resulting search is:
          https://www.google.com/#hl=en&sclie … mp;bih=576

          1. wilderness profile image95
            wildernessposted 4 years ago in reply to this

            I generally search for the URL.  It's almost always on the first line, sometimes the second.  Either way, if it's there it is indexed.

          2. AMFredenburg profile image78
            AMFredenburgposted 4 years ago in reply to this

            Thanks, Matthew!

            1. Matthew Meyer profile image74
              Matthew Meyerposted 4 years ago in reply to this

              You are very welcome!

  2. Len Cannon profile image87
    Len Cannonposted 4 years ago

    They're unseen but not unvisted. Spiders can and will follow noindex links, they just won't put them on search engines.

  3. rebekahELLE profile image88
    rebekahELLEposted 4 years ago

    Bots are always crawling to find new and updated, fresh content. 


    When you check hub stats at the top of each hub, click Search and it displays when search engines last crawled that page.

    1. AMFredenburg profile image78
      AMFredenburgposted 4 years ago in reply to this

      Wow, thanks! A really valuable bit of info.

    2. agilitymach profile image98
      agilitymachposted 4 years ago in reply to this

      Rebekah,

      I'm aware that Google bots are always crawling the web.  And I went to the stats tab to find the bot had crawled, but it crawled a hub that had been deindexed.  I guess this must be normal, but I thought Google and other engines couldn't see our deindexed (unfeatured) hubs.  It's not important, however  I am curious.

      1. rebekahELLE profile image88
        rebekahELLEposted 4 years ago in reply to this

        A page has to be crawled in order for a bot to  'see' the noindex tag.

        Sometimes a no-indexed page may still show in results if a bot hasn't crawled your site since the tag was added.

        1. agilitymach profile image98
          agilitymachposted 4 years ago in reply to this

          Thank you.  That is helpful.

  4. profile image0
    Jane Holmesposted 4 years ago

    That is surprising. It's good to know that it can be happening, but still a bit confusing.

    1. AMFredenburg profile image78
      AMFredenburgposted 4 years ago in reply to this

      Janie, did you check your hubs at the top of the page under "stats"? I didn't know my Hubs had been crawled, but I guess they are.

      1. profile image0
        Jane Holmesposted 4 years ago in reply to this

        Don't see that this has happened to any of my hubs to date. It is certainly a concern.

        1. AMFredenburg profile image78
          AMFredenburgposted 4 years ago in reply to this

          Have you signed up for Google Plus? If you get a gmail account you can sign up from there. That might help. There's also a "google authorship" hub floating around that I'll find, and I'll send you the url.

        2. rebekahELLE profile image88
          rebekahELLEposted 4 years ago in reply to this

          Jane, I see you've only been here for 2 months.  Sometimes it takes a while for a page to get ranked and to receive frequent crawls.

          I looked at your profile page and see that you have some niche topics that you're writing about.  I highly recommend linking your hubs together.  You can select a keyword, like 'sled dog' in your hubs and insert a link to another 'sled dog' hub.  You can link your berry articles, and your pet therapy articles.  Linking your sub-domain will help and it also helps with establishing relevancy and authority.

          This is a link to a series of articles that I found helpful trying to understand how search engines work. http://www.seomoz.org/beginners-guide-t … es-operate

          And yet with the nature of the giant beast that the internet has become, search is always evolving.

  5. Randy Godwin profile image92
    Randy Godwinposted 4 years ago

    Essentially, the bot sees nothing important if it is idled.

    1. agilitymach profile image98
      agilitymachposted 4 years ago in reply to this

      OK.  So it will crawl, see nothing, go away and not "index" the hub.  But it can still crawl the hub.

      Thanks!

    2. Alastar Packer profile image82
      Alastar Packerposted 4 years ago in reply to this

      Now that I've had some hubs have the no-follow tag when first crawled I'm beginning to suspect even when the hubs are finally indexed they're not treated the same as if the no-follow tag had never been seen on the hub to begin with. I really hope HP can soon get the vetting process done quicker for proven writers before the bots see a noindex-tag on a hub.

      1. NateB11 profile image93
        NateB11posted 4 years ago in reply to this

        This has concerned me a bit too. What I've been wondering, too, is if a hub has already been indexed, then gets idled and the no-follow tag, then gets crawled, if this affects indexing in anyway or is treated differently.

  6. Randy Godwin profile image92
    Randy Godwinposted 4 years ago

    But.....and there is a but, if it sees the no index tag it may not care to come back for a long while after the hub is edited.  Also, some people---and I am included in this camp--suspect this tag has a long term detrimental effect on the particular hub. 

    Most of the time when a hub is idled it never gets much Google traffic ever again.  Thus all of the controversy surrounding the new program.  Not to mention when a hub is idled you lose any organic back links which the hub has gained during its featured phase.  Too many possible bad things versus an unknown good, in my opinion.

    1. agilitymach profile image98
      agilitymachposted 4 years ago in reply to this

      Thank you.  That helps me understand why people get upset when a hub goes idle.  Perhaps there is a way to give the hub a whole new address, if the hubber so desires, to make Google look at it as "new."  Back links will still be lost, however.

    2. NateB11 profile image93
      NateB11posted 4 years ago in reply to this

      Yes, I've been concerned about this too. A hub could be already indexed, get idled and the no-follow tag, then get crawled. And I've been wondering what effect that has on the hub. I've wondered if it affected indexing. This seems especially relevant since it has been said that a hub (or any content) must age to start getting indexed and do well in search results, etc.

      Edit: I think it would help to know exactly what effect the no-follow tag has; it means it doesn't index. Then is it that it is seen as something not to index, by crawlers? And does it affect indexing that's already been done? If so, it would seem like erasing work done and age gained.

  7. daisydayz profile image87
    daisydayzposted 4 years ago

    I thought it just meant they were still crawled but not indexed in searches

  8. Randy Godwin profile image92
    Randy Godwinposted 4 years ago

    I've been waiting 21 hours now for my latest hub to be featured.  So much for the BS about times getting shorter.  And Simone wonders why they have the rep they do for not being trustworthy. DUH!

    1. agilitymach profile image98
      agilitymachposted 4 years ago in reply to this

      I've been waiting about 18 or 19 hours or so for my latest hub to get featured.  It got crawled by Google during that time, and it had a no index tag on it.

      However, I also posted the hub up on FB from which most of my traffic comes.  As this hub is going to be a "timely" hub and not evergreen, I needed it up and running quickly.  Google hits will matter little to nothing for this particular hub.  It should get hits from being "passed around" the target community. 

      That being said, if this hub were "timely" to a news story that I needed Google hits off of, it would have failed at this point.  I'm starting to see why some people worry about the long process for the QAP.  I honestly understand the need for a QA program after reading a lot of poor hubs, but I am hoping HP can come up with something a bit faster for its good writers.  (BTW, I'm honestly not sure I qualify for the "good writers" category, but I know others that would. smile)

  9. Glenn Stok profile image95
    Glenn Stokposted 4 years ago

    Based on the nature of this thread, dealing with Google crawling unindexed Hubs, I thought I'd add some interesting additional information.

    As was correctly explained by several people above, Google needs to keep crawling to check if and when the noindex tag is removed so that it can index the hub again.

    But I have discovered that Google also keeps crawling deleted URLs as well. How do I know this? Because I have my own business site where I can play with my own code.

    I wrote code in my 404 error page that shows me all the traffic of people (or bots) looking for non-existent pages. My report also shows me exactly what they are looking for -- the exact URL.

    I usually find hackers this way. They are looking for back doors to log in and do harm. I can then block them.   

    But the point I want to make here is that I also see Google bots crawling pages that I had deleted -- even years ago. For some reason, I guess, Google finds it important to keep checking if an old deleted page will ever be recreated.

    So, the bottom line is that Google will keep crawling anything that ever existed, if it was set to "noindex" or even if it was entirely deleted.

    Of course, I am not addressing the big issue -- a delay that may be imposed by noindexing. This all depends on how long it takes for Google to re-crawl, and then how long it takes for them to re-index after finding the that the noindex tag was removed. That timing all seems to be different with each website.  And maybe even with each subdomain.

    1. agilitymach profile image98
      agilitymachposted 4 years ago in reply to this

      That is interesting.  Google bots seem quite persistent. smile

 
working