Robots.txt file blocking URL

1. 73

   Quilligrapherposted 14 years ago

   Google AdSense Site Diagnostics reports that a Robots.txt file is blocking their crawler from accessing one of my hubs. Can anyone suggest how I can resolve this issue?

   replyreport
2. 60

   mel22posted 14 years ago

   check to see if that particular hub has the checkbox marked that says this hub conforms to google adsense policies. Should be in the field where you set the settings for ads to high low or none and where you change the title or toggle konterra on hubs. Not sure if thats it but its worth a shot to check

   replyreport
   1. 73

      Quilligrapherposted 14 years agoin reply to this

      Thanks for your prompt reply.
      
      Show Kontera Textlink Ads on this Hub? NOT CHECKED
       
      content meets the conditions of the Google AdSense Program Policy                         CHECKED
      
      this hub may be considered commerical   CHECKED
      
      These selections are the same for all of my hubs.

      replyreport
3. 60

   mel22posted 14 years ago

   I do know that those robots . txt thingies show up in the metadata which is searched by search engines so the question will have to be answered by someone more proficient in HTML I suppose. I always thought they had something to do with no follow links but since your author score is way above 75 I don't see how that would block you.  I'll leave this one alone and let someone else answer but I would be interested to know the hub url or title or a link, so I can look at the source code and see where that bugger is placed and why ,but i'm not promising an answer.

   replyreport
   1. 0

      shazwellynposted 14 years agoin reply to this

      Does below 75 on the hubscore not authorscore, have the same nofollow links?

      replyreport
4. 60

   mel22posted 14 years ago

   http://www.robotstxt.org/ this might give you some insight if you don't mind a little technical talk or reading i mean

   replyreport
5. 60

   mel22posted 14 years ago

   better yet http://www.google.com/support/webmaster … wer=156449
   
   seems you have a robots,txt file somehow embedded in your meta data of the hub.. the googlebot cannot access your hub until its removed.. basically the .txt file tells all crawlers not to crawl the site. in a text directive which all good crawlers heed the directive in the txt file.  Some spammers use crawlewrs that do not heed the .txt file and grab passwrds , etc., but googlebot is a well respected company and their boit heeds the directive in this robots.txt file. The question is how to remove it from the metadata of your hub. You may need to contact staff on this since they have access to change the source code. Was your hub ever flagged by staff recently?

   replyreport
   1. 73

      Quilligrapherposted 14 years agoin reply to this

      Many thanks for your suggestions, Mel.  I am looking into them now.

      replyreport
6. 62

   Mishaposted 14 years ago

   robots.txt is not embedded anywhere, it is located in the root of the site and controlled by site admins. Send email to the team

   replyreport
7. 98

   Glenn Stokposted 14 years ago

   Since a couple of months ago I have seen reports about HubPages' robots.txt file blocking Hubs. I was surprised to see it is still going on. I am a webmaster myself and I do my own HTML programming of my site. I can assure you that HubPages does not block individual Hubs.  All you have to do is open the robots.txt file in your browser and you'll see it is very innocent. It can always be found in the root directory of any complete website like HubPages.com
   
   As for the meta tags, HubPages places the description meta tag in each hub with the exact text that you enter as the summary of your hub. You are in control of that.
   
   When Google AdSense Site Diagnostics reports that a Robots.txt file is blocking their crawler from accessing one of your hubs, it may be due to a number of reasons not related to HubPages. Either someone clicked on the translate link in Google, or an archived copy of your hub has been residing somewhere. Both of these cases causes a copy of your hub to exist on another server and that server may have a robots.txt file blocking because they don't want the copy to be picked up by search engines. The problem is that your AdSence code is still in the copy, and that triggers the Site Diagnostics report.
   
   Bottom line, I am sure your original Hub is safe and your are getting credit for it. I already experienced a Hub of mine in the Diagnostics. Give it a few days and it clears itself up. This is not a result of anything HUbPages does.

   replyreport
   1. 60

      Kmadhavposted 14 years agoin reply to this

      Very good answer ....satisfied ....

      replyreport
8. 73

   Quilligrapherposted 14 years ago

   Thank you, Misha.  Thank you, Glenn.  I appreciate you both taking the time to offer your advice.
   
   The Google code defining the problem follows.  It does indeed contain a “translate” URL and flag:
   
   “http:/ / translate. googleusercontent. com/ translate_c? hl= pl&sl= en&u= http:/ / hubpages. com/ hub/ Sendler&prev= / search%3Fq%3Dstefan%2Bzgrzebski%26hl%3Dpl%26lr%3D&rurl= translate. google. pl&usg= ALkJrhg5VqTHa1TEYu9a1BFQbQ41KK0uhg”
   
   Does this support the suggestion that I give this issue a few days to vaporize itself?  I was reluctant to contact Hubpages Tech Support because I reasoned that they probably had enough AdSense savvy not to cause an issue of this sort.  The last crawl attempt was dated Jan. 02, 2010.  Should this blockage problem go away in a week or so?
   
   Again, my sincere thanks for your input.

   replyreport
   1. 71

      sunforgedposted 14 years agoin reply to this

      This issue has been responded to by hub admin in the past. I think ryankett started the thread.
      
      I believe is was explained as a non-issue
      
      The code you posted shows an event where a user used the "google translate" service to read your content. Since adsense is only allowed on pages that are written in english, it would only make sense that it was blocked from being crawled.
      
      
      other forum posts with this issue
      
      http://hubpages.com/search/include:forums+robots

      replyreport
9. 98

   Glenn Stokposted 14 years ago

   Quilligrapher, yes it should clear up in a week. I guessed right about it being translated Actually, I had it happen to me and the diagnostic report had cleared the next time I checked in about a week. Please remember that it is ONLY the translated copy that is blocked, not your original hub.
   
   sunforged, thanks for adding the additional info about why translations are blocked.

   replyreport
   1. 0

      shazwellynposted 14 years agoin reply to this

      you are always informative dark s x

      replyreport
   2. 73

      Quilligrapherposted 14 years agoin reply to this

      I am grateful for your input sunforged. I expect the Google AdSense blocked URL comment to vanish in a few days.
      
      Again, I appreciate your taking an interest in my thread, Glenn.  I intend to "pay it forward."

      replyreport
10. 0

    cosetteposted 14 years ago

    i get this too sometimes. the latest one says:
    
    http:/ / www. 123people. com/ ext/ frm? ti= person%20finder&search_term= laura%20buxton&search_country= US&st= person%20finder&target_url= http%3A%2F%2Fsnipsly. com%2F2009%2F12%2F09%2Fplease-return-to-laura-buxton%2F§ion= blog&wrt_id= 262 Robots.txt File Jan 4, 2010
    
    makes absolutely no sense to me, since i am not at 123people or anything, and i don't use snipsly either. they just go away and then other hubs get "blocked", but it's usually only two or three. if it was all of my hubs, then i would wonder.

    replyreport