What does it mean when Google webmaster says restricted by robots.txt?

1. 80

   tlpoagueposted 12 years ago

   I tried to asked this question before, but it didn't show up. I logged into Google Webmaster Tools and it said restricted by robots.txt. What is that and how do I fix it?
   It looks something like this:
   Show URLs: Restricted by robots.txt ‎(5)‎
     URL Detail Detected
   htt​p​/tl​poa​gue​.hu​bpa​ges​.co​m/x​ml/​com​men​t.p​hp  URL restricted by robots.txt Aug 23, 2011
   htt​p​/tl​poa​gue​.hu​bpa​ges​.co​m/h​ubt​ivi​ty  URL restricted by robots.txt Aug 22, 2011
   htt​p​/tl​poa​gue​.hu​bpa​ges​.co​m/w​idg​et/​tlp​oag​ue  URL restricted by robots.txt Aug 20, 2011
   htt​p​/tl​poa​gue​.hu​bpa​ges​.co​m/c​ont​act  URL restricted by robots.txt Aug 20, 2011
   htt​p​/tl​poa​gue​.hu​bpa​ges​.co​m/g​etw​idg​et  URL restricted by robots.txt Aug 18, 2011
   Any help would be greatly appreciated. Thanks!

   replyreport
   1. 87

      IzzyMposted 12 years agoin reply to this

      I got that as well! Am interested to know the answer.

      replyreport
2. 80

   tlpoagueposted 12 years ago

   Thanks Izzy, I think everyone is lost in the summaries. I guess I will have to check back later.

   replyreport
3. 75

   sofsposted 12 years ago

   I have got that too.. and that is when the plunge started... and I am still down.. what does that mean.. can someone explain and help us set it right ..please

   replyreport
4. 61

   CASE1WORKERposted 12 years ago

   What are URLs restricted by robots.txt errors?
   
   Google was unable to crawl the URL due to a robots.txt restriction. This can happen for a number of reasons. For instance, your robots.txt file might prohibit the Googlebot entirely; it might prohibit access to the directory in which this URL is located or it might prohibit access to the URL specifically. This is often not an error. You may have specifically set up a robots.txt file to prevent us from crawling this URL. If that is the case, there is no need to fix this; we will continue to respect robots.txt for this file.
   
   If a URL redirects to a URL that is blocked by a robots.txt file, the first URL will be reported as being blocked by robots.txt (even if the URL is listed as Allowed in the robots.txt analysis tool).
   
   
   Just cut this off the gooogle webmaster
   I think it expalins the traffic plunge surely as it says it prevents google crawling the URL?

   replyreport
5. 61

   CASE1WORKERposted 12 years ago

   MAYBE NOT- HERE IS A GOOGLE EXPLANATION
   http://alittlesearch4abigshare.blogspot … lways.html

   replyreport
6. 65

   2uesdayposted 12 years ago

   I too have this appearing for three hubs. I read it and tried to discover what I am meant to do about it but I cannot find an explanation that I can understand.
   
   How do you unblock it?
   Now it sounds like a plumbing problem not a search one

   replyreport
7. 60

   Urt og Dyrposted 12 years ago

   Don't worry about it. Here's the FAQ entry:
   http://hubpages.com/faq/#adsenserobots

   replyreport
   1. 65

      2uesdayposted 12 years agoin reply to this

      That is really helpful thank you, I never thought of looking for it there I had been searching Google related items.

      replyreport
8. 75

   sofsposted 12 years ago

   Thank you Case1worker and Urt og Dyr..
   but this keeps coming up every time since the subdomain change, never before that.. I seriously wonder if there is something to be done..

   replyreport
   1. 60

      Urt og Dyrposted 12 years agoin reply to this

      I hope a staff member will answer you. As far as I know, the FAQ entry itself was created in response to the many forum posts inquiring about this issue.
      
      http://hubpages.com/forum/topic/76611
      http://hubpages.com/forum/topic/32255
      http://hubpages.com/forum/topic/29792
      http://hubpages.com/forum/topic/26822
      http://hubpages.com/forum/topic/37861
      http://hubpages.com/forum/topic/34945
      http://hubpages.com/forum/topic/33803
      http://hubpages.com/forum/topic/27939
      and so on

      replyreport
      1. 75

         sofsposted 12 years agoin reply to this

         Thank you so much...I am reading all this though it sounds like Greek and Latin to me I appreciate the effort that you have taken greatly..

         replyreport
9. 80

   tlpoagueposted 12 years ago

   Thanks for the information. This helps! I was freaking out that I had screwed something up again.

   replyreport
   1. 75

      sofsposted 12 years agoin reply to this

      It just says that we don't have to do anything.. . I  just left it at that..

      replyreport