Unindexed Hub Listed as Having a "Google Crawl"

1. 91

   agilitymachposted 10 years ago

   I have a hub that went unfeatured several days back.  It was my second hub, and it really was in dire need of rewriting.  After doing an extensive revision, I sent it out to the QAP.  Then I checked the stats to see how many views it has gotten to date.
   
   I was surprised to see that a Google bot had crawled it two hours ago, and a Bing bot had crawled it 12 hours ago.   Aren't unfeatured hubs supposed to be "unseen" by the search engines? 
   
   Any idea why the bot crawls?  This isn't a huge issue, but I am curious.  The hub is now listed as "pending."

   replyreport
   1. 74

      AMFredenburgposted 10 years agoin reply to this

      How could you tell that the bots were crawling?

      replyreport
   2. 73

      Matthew Meyerposted 10 years agoin reply to this

      Crawling and being indexed in a search engine are different things.

      replyreport
      1. 74

         AMFredenburgposted 10 years agoin reply to this

         Okay, I am being crawled; how do I find out if I am indexed?

         replyreport
         1. 73

            Matthew Meyerposted 10 years agoin reply to this

            By searching for your Hub in the Search Engine in which you would like to determine indexed status.
            
            So if I was trying to see if this Hub is indexed in Google:
            http://hubpages.com/learningcenter/Diag … r-problems
            
            You would search this exact string in Google:
            "site:hubpages.com/learningcenter/Diagnosing-common-browser-problems/"
            
            So the resulting search is:
            https://www.google.com/#hl=en&sclie … mp;bih=576

            replyreport
            1. 95

               wildernessposted 10 years agoin reply to this

               I generally search for the URL.  It's almost always on the first line, sometimes the second.  Either way, if it's there it is indexed.

               replyreport
            2. 74

               AMFredenburgposted 10 years agoin reply to this

               Thanks, Matthew!

               replyreport
               1. 73

                  Matthew Meyerposted 10 years agoin reply to this

                  You are very welcome!

                  replyreport
2. 88

   Len Cannonposted 10 years ago

   They're unseen but not unvisted. Spiders can and will follow noindex links, they just won't put them on search engines.

   replyreport
3. 85

   rebekahELLEposted 10 years ago

   Bots are always crawling to find new and updated, fresh content. 
   
   
   When you check hub stats at the top of each hub, click Search and it displays when search engines last crawled that page.

   replyreport
   1. 74

      AMFredenburgposted 10 years agoin reply to this

      Wow, thanks! A really valuable bit of info.

      replyreport
   2. 91

      agilitymachposted 10 years agoin reply to this

      Rebekah,
      
      I'm aware that Google bots are always crawling the web.  And I went to the stats tab to find the bot had crawled, but it crawled a hub that had been deindexed.  I guess this must be normal, but I thought Google and other engines couldn't see our deindexed (unfeatured) hubs.  It's not important, however  I am curious.

      replyreport
      1. 85

         rebekahELLEposted 10 years agoin reply to this

         A page has to be crawled in order for a bot to  'see' the noindex tag.
         
         Sometimes a no-indexed page may still show in results if a bot hasn't crawled your site since the tag was added.

         replyreport
         1. 91

            agilitymachposted 10 years agoin reply to this

            Thank you.  That is helpful.

            replyreport
4. 0

   Jane Holmesposted 10 years ago

   That is surprising. It's good to know that it can be happening, but still a bit confusing.

   replyreport
   1. 74

      AMFredenburgposted 10 years agoin reply to this

      Janie, did you check your hubs at the top of the page under "stats"? I didn't know my Hubs had been crawled, but I guess they are.

      replyreport
      1. 0

         Jane Holmesposted 10 years agoin reply to this

         Don't see that this has happened to any of my hubs to date. It is certainly a concern.

         replyreport
         1. 74

            AMFredenburgposted 10 years agoin reply to this

            Have you signed up for Google Plus? If you get a gmail account you can sign up from there. That might help. There's also a "google authorship" hub floating around that I'll find, and I'll send you the url.

            replyreport
         2. 85

            rebekahELLEposted 10 years agoin reply to this

            Jane, I see you've only been here for 2 months.  Sometimes it takes a while for a page to get ranked and to receive frequent crawls.
            
            I looked at your profile page and see that you have some niche topics that you're writing about.  I highly recommend linking your hubs together.  You can select a keyword, like 'sled dog' in your hubs and insert a link to another 'sled dog' hub.  You can link your berry articles, and your pet therapy articles.  Linking your sub-domain will help and it also helps with establishing relevancy and authority.
            
            This is a link to a series of articles that I found helpful trying to understand how search engines work. http://www.seomoz.org/beginners-guide-t … es-operate
            
            And yet with the nature of the giant beast that the internet has become, search is always evolving.

            replyreport
5. 61

   Randy Godwinposted 10 years ago

   Essentially, the bot sees nothing important if it is idled.

   replyreport
   1. 91

      agilitymachposted 10 years agoin reply to this

      OK.  So it will crawl, see nothing, go away and not "index" the hub.  But it can still crawl the hub.
      
      Thanks!

      replyreport
   2. 73

      Alastar Packerposted 10 years agoin reply to this

      Now that I've had some hubs have the no-follow tag when first crawled I'm beginning to suspect even when the hubs are finally indexed they're not treated the same as if the no-follow tag had never been seen on the hub to begin with. I really hope HP can soon get the vetting process done quicker for proven writers before the bots see a noindex-tag on a hub.

      replyreport
      1. 89

         NateB11posted 10 years agoin reply to this

         This has concerned me a bit too. What I've been wondering, too, is if a hub has already been indexed, then gets idled and the no-follow tag, then gets crawled, if this affects indexing in anyway or is treated differently.

         replyreport
6. 61

   Randy Godwinposted 10 years ago

   But.....and there is a but, if it sees the no index tag it may not care to come back for a long while after the hub is edited.  Also, some people---and I am included in this camp--suspect this tag has a long term detrimental effect on the particular hub. 
   
   Most of the time when a hub is idled it never gets much Google traffic ever again.  Thus all of the controversy surrounding the new program.  Not to mention when a hub is idled you lose any organic back links which the hub has gained during its featured phase.  Too many possible bad things versus an unknown good, in my opinion.

   replyreport
   1. 91

      agilitymachposted 10 years agoin reply to this

      Thank you.  That helps me understand why people get upset when a hub goes idle.  Perhaps there is a way to give the hub a whole new address, if the hubber so desires, to make Google look at it as "new."  Back links will still be lost, however.

      replyreport
   2. 89

      NateB11posted 10 years agoin reply to this

      Yes, I've been concerned about this too. A hub could be already indexed, get idled and the no-follow tag, then get crawled. And I've been wondering what effect that has on the hub. I've wondered if it affected indexing. This seems especially relevant since it has been said that a hub (or any content) must age to start getting indexed and do well in search results, etc.
      
      Edit: I think it would help to know exactly what effect the no-follow tag has; it means it doesn't index. Then is it that it is seen as something not to index, by crawlers? And does it affect indexing that's already been done? If so, it would seem like erasing work done and age gained.

      replyreport
7. 83

   daisydayzposted 10 years ago

   I thought it just meant they were still crawled but not indexed in searches

   replyreport
8. 61

   Randy Godwinposted 10 years ago

   I've been waiting 21 hours now for my latest hub to be featured.  So much for the BS about times getting shorter.  And Simone wonders why they have the rep they do for not being trustworthy. DUH!

   replyreport
   1. 91

      agilitymachposted 10 years agoin reply to this

      I've been waiting about 18 or 19 hours or so for my latest hub to get featured.  It got crawled by Google during that time, and it had a no index tag on it.
      
      However, I also posted the hub up on FB from which most of my traffic comes.  As this hub is going to be a "timely" hub and not evergreen, I needed it up and running quickly.  Google hits will matter little to nothing for this particular hub.  It should get hits from being "passed around" the target community. 
      
      That being said, if this hub were "timely" to a news story that I needed Google hits off of, it would have failed at this point.  I'm starting to see why some people worry about the long process for the QAP.  I honestly understand the need for a QA program after reading a lot of poor hubs, but I am hoping HP can come up with something a bit faster for its good writers.  (BTW, I'm honestly not sure I qualify for the "good writers" category, but I know others that would. )

      replyreport
9. 96

   Glenn Stokposted 10 years ago

   Based on the nature of this thread, dealing with Google crawling unindexed Hubs, I thought I'd add some interesting additional information.
   
   As was correctly explained by several people above, Google needs to keep crawling to check if and when the noindex tag is removed so that it can index the hub again.
   
   But I have discovered that Google also keeps crawling deleted URLs as well. How do I know this? Because I have my own business site where I can play with my own code.
   
   I wrote code in my 404 error page that shows me all the traffic of people (or bots) looking for non-existent pages. My report also shows me exactly what they are looking for -- the exact URL.
   
   I usually find hackers this way. They are looking for back doors to log in and do harm. I can then block them.   
   
   But the point I want to make here is that I also see Google bots crawling pages that I had deleted -- even years ago. For some reason, I guess, Google finds it important to keep checking if an old deleted page will ever be recreated.
   
   So, the bottom line is that Google will keep crawling anything that ever existed, if it was set to "noindex" or even if it was entirely deleted.
   
   Of course, I am not addressing the big issue -- a delay that may be imposed by noindexing. This all depends on how long it takes for Google to re-crawl, and then how long it takes for them to re-index after finding the that the noindex tag was removed. That timing all seems to be different with each website.  And maybe even with each subdomain.

   replyreport
   1. 91

      agilitymachposted 10 years agoin reply to this

      That is interesting.  Google bots seem quite persistent.

      replyreport