Unindexed Hub Listed as Having a "Google Crawl"

Jump to Last Post 1-9 of 9 discussions (31 posts)
  1. agilitymach profile image98
    agilitymachposted 5 years ago

    I have a hub that went unfeatured several days back.  It was my second hub, and it really was in dire need of rewriting.  After doing an extensive revision, I sent it out to the QAP.  Then I checked the stats to see how many views it has gotten to date.

    I was surprised to see that a Google bot had crawled it two hours ago, and a Bing bot had crawled it 12 hours ago.   Aren't unfeatured hubs supposed to be "unseen" by the search engines? 

    Any idea why the bot crawls?  This isn't a huge issue, but I am curious.  The hub is now listed as "pending."

    1. AMFredenburg profile image78
      AMFredenburgposted 5 years agoin reply to this

      How could you tell that the bots were crawling?

    2. Matthew Meyer profile image73
      Matthew Meyerposted 5 years agoin reply to this

      Crawling and being indexed in a search engine are different things.

      1. AMFredenburg profile image78
        AMFredenburgposted 5 years agoin reply to this

        Okay, I am being crawled; how do I find out if I am indexed?

        1. Matthew Meyer profile image73
          Matthew Meyerposted 5 years agoin reply to this

          By searching for your Hub in the Search Engine in which you would like to determine indexed status.

          So if I was trying to see if this Hub is indexed in Google:
          http://hubpages.com/learningcenter/Diag … r-problems

          You would search this exact string in Google:
          "site:hubpages.com/learningcenter/Diagnosing-common-browser-problems/"

          So the resulting search is:
          https://www.google.com/#hl=en&sclie … mp;bih=576

          1. wilderness profile image98
            wildernessposted 5 years agoin reply to this

            I generally search for the URL.  It's almost always on the first line, sometimes the second.  Either way, if it's there it is indexed.

          2. AMFredenburg profile image78
            AMFredenburgposted 5 years agoin reply to this

            Thanks, Matthew!

            1. Matthew Meyer profile image73
              Matthew Meyerposted 5 years agoin reply to this

              You are very welcome!

  2. Len Cannon profile image89
    Len Cannonposted 5 years ago

    They're unseen but not unvisted. Spiders can and will follow noindex links, they just won't put them on search engines.

  3. rebekahELLE profile image88
    rebekahELLEposted 5 years ago

    Bots are always crawling to find new and updated, fresh content. 


    When you check hub stats at the top of each hub, click Search and it displays when search engines last crawled that page.

    1. AMFredenburg profile image78
      AMFredenburgposted 5 years agoin reply to this

      Wow, thanks! A really valuable bit of info.

    2. agilitymach profile image98
      agilitymachposted 5 years agoin reply to this

      Rebekah,

      I'm aware that Google bots are always crawling the web.  And I went to the stats tab to find the bot had crawled, but it crawled a hub that had been deindexed.  I guess this must be normal, but I thought Google and other engines couldn't see our deindexed (unfeatured) hubs.  It's not important, however  I am curious.

      1. rebekahELLE profile image88
        rebekahELLEposted 5 years agoin reply to this

        A page has to be crawled in order for a bot to  'see' the noindex tag.

        Sometimes a no-indexed page may still show in results if a bot hasn't crawled your site since the tag was added.

        1. agilitymach profile image98
          agilitymachposted 5 years agoin reply to this

          Thank you.  That is helpful.

  4. profile image0
    Jane Holmesposted 5 years ago

    That is surprising. It's good to know that it can be happening, but still a bit confusing.

    1. AMFredenburg profile image78
      AMFredenburgposted 5 years agoin reply to this

      Janie, did you check your hubs at the top of the page under "stats"? I didn't know my Hubs had been crawled, but I guess they are.

      1. profile image0
        Jane Holmesposted 5 years agoin reply to this

        Don't see that this has happened to any of my hubs to date. It is certainly a concern.

        1. AMFredenburg profile image78
          AMFredenburgposted 5 years agoin reply to this

          Have you signed up for Google Plus? If you get a gmail account you can sign up from there. That might help. There's also a "google authorship" hub floating around that I'll find, and I'll send you the url.

        2. rebekahELLE profile image88
          rebekahELLEposted 5 years agoin reply to this

          Jane, I see you've only been here for 2 months.  Sometimes it takes a while for a page to get ranked and to receive frequent crawls.

          I looked at your profile page and see that you have some niche topics that you're writing about.  I highly recommend linking your hubs together.  You can select a keyword, like 'sled dog' in your hubs and insert a link to another 'sled dog' hub.  You can link your berry articles, and your pet therapy articles.  Linking your sub-domain will help and it also helps with establishing relevancy and authority.

          This is a link to a series of articles that I found helpful trying to understand how search engines work. http://www.seomoz.org/beginners-guide-t … es-operate

          And yet with the nature of the giant beast that the internet has become, search is always evolving.

  5. Randy Godwin profile image91
    Randy Godwinposted 5 years ago

    Essentially, the bot sees nothing important if it is idled.

    1. agilitymach profile image98
      agilitymachposted 5 years agoin reply to this

      OK.  So it will crawl, see nothing, go away and not "index" the hub.  But it can still crawl the hub.

      Thanks!

    2. Alastar Packer profile image81
      Alastar Packerposted 5 years agoin reply to this

      Now that I've had some hubs have the no-follow tag when first crawled I'm beginning to suspect even when the hubs are finally indexed they're not treated the same as if the no-follow tag had never been seen on the hub to begin with. I really hope HP can soon get the vetting process done quicker for proven writers before the bots see a noindex-tag on a hub.

      1. NateB11 profile image92
        NateB11posted 5 years agoin reply to this

        This has concerned me a bit too. What I've been wondering, too, is if a hub has already been indexed, then gets idled and the no-follow tag, then gets crawled, if this affects indexing in anyway or is treated differently.

  6. Randy Godwin profile image91
    Randy Godwinposted 5 years ago

    But.....and there is a but, if it sees the no index tag it may not care to come back for a long while after the hub is edited.  Also, some people---and I am included in this camp--suspect this tag has a long term detrimental effect on the particular hub. 

    Most of the time when a hub is idled it never gets much Google traffic ever again.  Thus all of the controversy surrounding the new program.  Not to mention when a hub is idled you lose any organic back links which the hub has gained during its featured phase.  Too many possible bad things versus an unknown good, in my opinion.

    1. agilitymach profile image98
      agilitymachposted 5 years agoin reply to this

      Thank you.  That helps me understand why people get upset when a hub goes idle.  Perhaps there is a way to give the hub a whole new address, if the hubber so desires, to make Google look at it as "new."  Back links will still be lost, however.

    2. NateB11 profile image92
      NateB11posted 5 years agoin reply to this

      Yes, I've been concerned about this too. A hub could be already indexed, get idled and the no-follow tag, then get crawled. And I've been wondering what effect that has on the hub. I've wondered if it affected indexing. This seems especially relevant since it has been said that a hub (or any content) must age to start getting indexed and do well in search results, etc.

      Edit: I think it would help to know exactly what effect the no-follow tag has; it means it doesn't index. Then is it that it is seen as something not to index, by crawlers? And does it affect indexing that's already been done? If so, it would seem like erasing work done and age gained.

  7. daisydayz profile image87
    daisydayzposted 5 years ago

    I thought it just meant they were still crawled but not indexed in searches

  8. Randy Godwin profile image91
    Randy Godwinposted 5 years ago

    I've been waiting 21 hours now for my latest hub to be featured.  So much for the BS about times getting shorter.  And Simone wonders why they have the rep they do for not being trustworthy. DUH!

    1. agilitymach profile image98
      agilitymachposted 5 years agoin reply to this

      I've been waiting about 18 or 19 hours or so for my latest hub to get featured.  It got crawled by Google during that time, and it had a no index tag on it.

      However, I also posted the hub up on FB from which most of my traffic comes.  As this hub is going to be a "timely" hub and not evergreen, I needed it up and running quickly.  Google hits will matter little to nothing for this particular hub.  It should get hits from being "passed around" the target community. 

      That being said, if this hub were "timely" to a news story that I needed Google hits off of, it would have failed at this point.  I'm starting to see why some people worry about the long process for the QAP.  I honestly understand the need for a QA program after reading a lot of poor hubs, but I am hoping HP can come up with something a bit faster for its good writers.  (BTW, I'm honestly not sure I qualify for the "good writers" category, but I know others that would. smile)

  9. Glenn Stok profile image99
    Glenn Stokposted 5 years ago

    Based on the nature of this thread, dealing with Google crawling unindexed Hubs, I thought I'd add some interesting additional information.

    As was correctly explained by several people above, Google needs to keep crawling to check if and when the noindex tag is removed so that it can index the hub again.

    But I have discovered that Google also keeps crawling deleted URLs as well. How do I know this? Because I have my own business site where I can play with my own code.

    I wrote code in my 404 error page that shows me all the traffic of people (or bots) looking for non-existent pages. My report also shows me exactly what they are looking for -- the exact URL.

    I usually find hackers this way. They are looking for back doors to log in and do harm. I can then block them.   

    But the point I want to make here is that I also see Google bots crawling pages that I had deleted -- even years ago. For some reason, I guess, Google finds it important to keep checking if an old deleted page will ever be recreated.

    So, the bottom line is that Google will keep crawling anything that ever existed, if it was set to "noindex" or even if it was entirely deleted.

    Of course, I am not addressing the big issue -- a delay that may be imposed by noindexing. This all depends on how long it takes for Google to re-crawl, and then how long it takes for them to re-index after finding the that the noindex tag was removed. That timing all seems to be different with each website.  And maybe even with each subdomain.

    1. agilitymach profile image98
      agilitymachposted 5 years agoin reply to this

      That is interesting.  Google bots seem quite persistent. smile

 
working

This website uses cookies

As a user in the EEA, your approval is needed on a few things. To provide a better website experience, hubpages.com uses cookies (and other similar technologies) and may collect, process, and share personal data. Please choose which areas of our service you consent to our doing so.

For more information on managing or withdrawing consents and how we handle data, visit our Privacy Policy at: https://hubpages.com/privacy-policy#gdpr

Show Details
Necessary
HubPages Device IDThis is used to identify particular browsers or devices when the access the service, and is used for security reasons.
LoginThis is necessary to sign in to the HubPages Service.
Google RecaptchaThis is used to prevent bots and spam. (Privacy Policy)
AkismetThis is used to detect comment spam. (Privacy Policy)
HubPages Google AnalyticsThis is used to provide data on traffic to our website, all personally identifyable data is anonymized. (Privacy Policy)
HubPages Traffic PixelThis is used to collect data on traffic to articles and other pages on our site. Unless you are signed in to a HubPages account, all personally identifiable information is anonymized.
Amazon Web ServicesThis is a cloud services platform that we used to host our service. (Privacy Policy)
CloudflareThis is a cloud CDN service that we use to efficiently deliver files required for our service to operate such as javascript, cascading style sheets, images, and videos. (Privacy Policy)
Google Hosted LibrariesJavascript software libraries such as jQuery are loaded at endpoints on the googleapis.com or gstatic.com domains, for performance and efficiency reasons. (Privacy Policy)
Features
Google Custom SearchThis is feature allows you to search the site. (Privacy Policy)
Google MapsSome articles have Google Maps embedded in them. (Privacy Policy)
Google ChartsThis is used to display charts and graphs on articles and the author center. (Privacy Policy)
Google AdSense Host APIThis service allows you to sign up for or associate a Google AdSense account with HubPages, so that you can earn money from ads on your articles. No data is shared unless you engage with this feature. (Privacy Policy)
Google YouTubeSome articles have YouTube videos embedded in them. (Privacy Policy)
VimeoSome articles have Vimeo videos embedded in them. (Privacy Policy)
PaypalThis is used for a registered author who enrolls in the HubPages Earnings program and requests to be paid via PayPal. No data is shared with Paypal unless you engage with this feature. (Privacy Policy)
Facebook LoginYou can use this to streamline signing up for, or signing in to your Hubpages account. No data is shared with Facebook unless you engage with this feature. (Privacy Policy)
MavenThis supports the Maven widget and search functionality. (Privacy Policy)
Marketing
Google AdSenseThis is an ad network. (Privacy Policy)
Google DoubleClickGoogle provides ad serving technology and runs an ad network. (Privacy Policy)
Index ExchangeThis is an ad network. (Privacy Policy)
SovrnThis is an ad network. (Privacy Policy)
Facebook AdsThis is an ad network. (Privacy Policy)
Amazon Unified Ad MarketplaceThis is an ad network. (Privacy Policy)
AppNexusThis is an ad network. (Privacy Policy)
OpenxThis is an ad network. (Privacy Policy)
Rubicon ProjectThis is an ad network. (Privacy Policy)
TripleLiftThis is an ad network. (Privacy Policy)
Say MediaWe partner with Say Media to deliver ad campaigns on our sites. (Privacy Policy)
Remarketing PixelsWe may use remarketing pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to advertise the HubPages Service to people that have visited our sites.
Conversion Tracking PixelsWe may use conversion tracking pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to identify when an advertisement has successfully resulted in the desired action, such as signing up for the HubPages Service or publishing an article on the HubPages Service.
Statistics
Author Google AnalyticsThis is used to provide traffic data and reports to the authors of articles on the HubPages Service. (Privacy Policy)
ComscoreComScore is a media measurement and analytics company providing marketing data and analytics to enterprises, media and advertising agencies, and publishers. Non-consent will result in ComScore only processing obfuscated personal data. (Privacy Policy)
Amazon Tracking PixelSome articles display amazon products as part of the Amazon Affiliate program, this pixel provides traffic statistics for those products (Privacy Policy)