How to Increase Crawl Rates on Your Subdomain

1. 83

   Simone Smithposted 12 years ago

   Hey y'all!
   
   From time to time we hear concerns about newly-Featured Hubs not getting crawled for aaaaages, so we tried out a couple of different solutions (er- rather Paul Edmondson did; I can't take the credit) and finally found something that helps: increasing your subdomain's crawl rate through Google Webmaster Tools.
   
   I've added tips on doing this to our Learning Center guides to Google Webmaster Tools and Featured Hubs, but the how-to is all in this FAQ entry: http://hubpages.com/faq/#increase-crawl-rate
   
   Should you be concerned about the rate at which your Hubs are Featured, give this a go!

   replyreport
   1. 84

      vocalcoachposted 12 years agoin reply to this

      Thanks Simone.  This will be a big help.  Gonna go study your tips as well as the FAQ entry!

      replyreport
      1. 83

         Simone Smithposted 12 years agoin reply to this

         Sweet! This one is pretty straightforward. The only catch is that the increased crawl rate has to be re-set every 90 days!

         replyreport
         1. 73

            AMFredenburgposted 12 years agoin reply to this

            A valuable nugget of information; thanks!

            replyreport
         2. 82

            tillsontitanposted 12 years agoin reply to this

            Increased crawl rate has to be re-set every 90 days??? Can you explain that one Simone?

            replyreport
   2. 75

      shampa sadhyaposted 12 years agoin reply to this

      Hello Simone Smith,
      
      Thanks for this info but while I was fixing the crawl rate I noticed that 16 hubs of mine have URL errors. It suggests: 404 Page Not Found. I tried to fix 3 or 4 of them but not confident that I did it correctly. I am not sound in these type of technical matters. I simply need your help.
      
      May be this is one reason why my traffic has gone down.
      
      Even I have noticed that in 1 or 2 hub the URL which is mentioned is never used by me. Fact is, I have changed the title of many hubs but did not knew about this kind of error.
      
      I am feeling helpless so kindly help me to fix the error in a step by step method.   
      
      Thanks in advance!

      replyreport
      1. 83

         Simone Smithposted 12 years agoin reply to this

         Hey shampa sadhya- are you talking about error messages in Webmaster Tools, or on HubPages?
         
         Rest assured that when you change a title of a Hub, its URL does NOT change. Those cannot be edited once created.

         replyreport
   3. 75

      sholland10posted 12 years agoin reply to this

      Thanks, Simone!  I had some adjustments to make.  So glad I found your post! :-)

      replyreport
2. 0

   Ethan Greenposted 12 years ago

   That's interesting - I didn't know that was possible. Great advice for my own websites as well.
   Thank you Simone!
   Ethan

   replyreport
   1. 83

      Simone Smithposted 12 years agoin reply to this

      Oh, totally! Hope it helps!

      replyreport
3. 83

   EricDockettposted 12 years ago

   If we increase our crawl rate, wouldn't that also mean that we'd have a smaller window to edit our Hubs when they fall into idle status before Googlebot comes back and de-indexes them?

   replyreport
   1. 83

      Simone Smithposted 12 years agoin reply to this

      Yep! So it's all up to you. It definitely goes both ways.

      replyreport
   2. 75

      shampa sadhyaposted 12 years agoin reply to this

      What does smaller window mean?

      replyreport
      1. 83

         EricDockettposted 12 years agoin reply to this

         It means, if you increase your crawl rate, you'll have less time to edit and fix up your Hub when it falls into non-featured status before Google re-crawls your subdomain and de-indexes your Hub.

         replyreport
         1. 86

            Marisa Wrightposted 12 years agoin reply to this

            Good point.

            replyreport
4. 69

   LCDWriterposted 12 years ago

   Shampa, were those pages unfeatured?  (no H beside them).  That might be the reason they have an error.

   replyreport
   1. 75

      shampa sadhyaposted 12 years agoin reply to this

      No, they are all featured hubs. That's why I am very much anxious.

      replyreport
      1. 69

         LCDWriterposted 12 years agoin reply to this

         Very strange!

         replyreport
      2. 76

         wildernessposted 12 years agoin reply to this

         I wouldn't worry too much - I think that's not too unusual.  For whatever reason google couldn't follow a link from somewhere, so the page couldn't be found.  Maybe HP had a 2 second down time, maybe someone else did.  If often see a few of those, gone a few days later.
         
         If it continues, though, it could indicate something actually wrong.

         replyreport
         1. 83

            Simone Smithposted 12 years agoin reply to this

            That's what I'm thinking. I haven't really seen this happen much though (not that I spend much of any time looking at my Hubs through Webmaser Tools....)

            replyreport
5. 73

   AMFredenburgposted 12 years ago

   Well, I've done it; we'll see if there's a change. I've only got 4 Hubs so far, so perhaps it won't show all that much.

   replyreport
6. 68

   Rosie2010posted 12 years ago

   Great useful tip!  Thanks, Simone.

   replyreport
7. 61

   paradigmsearchposted 12 years ago

   I checked some hubs at random. Both ones Google loves and ones Google hates. The absolute oldest crawl date I could find was 3 days. I've decided that since it ain't broke, I'm not going to fix it. What with the oldest crawl date being only 3 days, this kinda indicates to me that Google actually likes my subdomain. Or am I living in fantasy land here?   

   replyreport
8. 79

   LuisEGonzalezposted 12 years ago

   I did as suggested by Simone and some of my hubs have started showing red arrows. Let's hope this was a good move..................

   replyreport
9. 83

   Marcy Goodfleischposted 12 years ago

   Thanks, Simone - I will go read and learn how to do this!

   replyreport
10. 78

    Li Galoposted 12 years ago

    Thanks for the helpful info.

    replyreport
11. 0

    Beth37posted 12 years ago

    How do you do this?:
    
    Navigate to your HubPages subdomain on Google Webmaster Tools

    replyreport
12. 0

    Confucius Vijayposted 12 years ago

    I think, to say to google to crawl the page, we should submit the RSS feed to google. as after feeding the rss google crawlr will be able to crawl your page fast.

    replyreport