jump to last post 1-9 of 9 discussions (14 posts)

What does it mean when Google webmaster says restricted by robots.txt?

  1. tlpoague profile image87
    tlpoagueposted 6 years ago

    I tried to asked this question before, but it didn't show up. I logged into Google Webmaster Tools and it said restricted by robots.txt. What is that and how do I fix it?
    It looks something like this:
    Show URLs: Restricted by robots.txt ‎(5)‎
      URL Detail Detected
    htt​phmm​/tl​poa​gue​.hu​bpa​ges​.co​m/x​ml/​com​men​t.p​hp  URL restricted by robots.txt Aug 23, 2011
    htt​phmm​/tl​poa​gue​.hu​bpa​ges​.co​m/h​ubt​ivi​ty  URL restricted by robots.txt Aug 22, 2011
    htt​phmm​/tl​poa​gue​.hu​bpa​ges​.co​m/w​idg​et/​tlp​oag​ue  URL restricted by robots.txt Aug 20, 2011
    htt​phmm​/tl​poa​gue​.hu​bpa​ges​.co​m/c​ont​act  URL restricted by robots.txt Aug 20, 2011
    htt​phmm​/tl​poa​gue​.hu​bpa​ges​.co​m/g​etw​idg​et  URL restricted by robots.txt Aug 18, 2011
    Any help would be greatly appreciated. Thanks!

    1. IzzyM profile image86
      IzzyMposted 6 years ago in reply to this

      I got that as well! Am interested to know the answer.

  2. tlpoague profile image87
    tlpoagueposted 6 years ago

    Thanks Izzy, I think everyone is lost in the summaries. I guess I will have to check back later.

  3. sofs profile image84
    sofsposted 6 years ago

    I have got that too.. and that is when the plunge started... and I am still down.. what does that mean.. can someone explain and help us set it right ..please

  4. CASE1WORKER profile image78
    CASE1WORKERposted 6 years ago

    What are URLs restricted by robots.txt errors?

    Google was unable to crawl the URL due to a robots.txt restriction. This can happen for a number of reasons. For instance, your robots.txt file might prohibit the Googlebot entirely; it might prohibit access to the directory in which this URL is located or it might prohibit access to the URL specifically. This is often not an error. You may have specifically set up a robots.txt file to prevent us from crawling this URL. If that is the case, there is no need to fix this; we will continue to respect robots.txt for this file.

    If a URL redirects to a URL that is blocked by a robots.txt file, the first URL will be reported as being blocked by robots.txt (even if the URL is listed as Allowed in the robots.txt analysis tool).


    Just cut this off the gooogle webmaster
    I think it expalins the traffic plunge surely as it says it prevents google crawling the URL?

  5. CASE1WORKER profile image78
    CASE1WORKERposted 6 years ago

    MAYBE NOT- HERE IS A GOOGLE EXPLANATION
    http://alittlesearch4abigshare.blogspot … lways.html

  6. 2uesday profile image84
    2uesdayposted 6 years ago

    I too have this appearing for three hubs. I read it and tried to discover what I am meant to do about it but I cannot find an explanation that I can understand.

    How do you unblock it?
    Now it sounds like a plumbing problem not a search one smile

  7. Urt og Dyr profile image60
    Urt og Dyrposted 6 years ago

    Don't worry about it. Here's the FAQ entry:
    http://hubpages.com/faq/#adsenserobots

    1. 2uesday profile image84
      2uesdayposted 6 years ago in reply to this

      That is really helpful thank you, I never thought of looking for it there I had been searching Google related items.

  8. sofs profile image84
    sofsposted 6 years ago

    Thank you Case1worker and Urt og Dyr..
    but this keeps coming up every time since the subdomain change, never before that.. I seriously wonder if there is something to be done..

    1. Urt og Dyr profile image60
      Urt og Dyrposted 6 years ago in reply to this
      1. sofs profile image84
        sofsposted 6 years ago in reply to this

        Thank you so much...I am reading all this though it sounds like Greek and Latin to me smile I appreciate the effort that you have taken greatly.. smile

  9. tlpoague profile image87
    tlpoagueposted 6 years ago

    Thanks for the information. This helps! I was freaking out that I had screwed something up again.

    1. sofs profile image84
      sofsposted 6 years ago in reply to this

      It just says that we don't have to do anything.. . I  just left it at that.. smile

 
working