Google Adsense blocked due to Robots.txt File

Jump to Last Post 1-31 of 31 discussions (47 posts)
  1. cashmere profile image82
    cashmereposted 9 years ago

    In my site diagnostics in Google adsense these hubs of mine are being shown as blocked. Reason blocked is Robots.txt File.
    Can you tell me how I can unblock my following hubs?

    My favourite Tarot layouts,
    How to teach English
    and
    Multilevel marketing - the good the bad and ugly.

    This has happened since Nov 5, but these hubs have been published months ago.
    What happened and how can I unblock them?

    1. profile image0
      Pacal Votanposted 9 years agoin reply to this

      Robots.txt is on hubpages server. It's where hubstaff specifies which pages search robots should not scan. For some reason they chose to hide these hubs of yours from search engines. You have no access to robots.txt so you can do nothing about it.

  2. cashmere profile image82
    cashmereposted 9 years ago

    Okay, but why would they suddenly choose to block my hubs now? Three months after publication?

  3. Marisa Wright profile image96
    Marisa Wrightposted 9 years ago

    I never even looked at site diagnostics before!  I just checked and four of my Hubs show as blocked, too.

    The Help screen suggests checking again later, as it may just be that the server was down when they tried to crawl the site.  Let's hope that's the reason.

  4. cashmere profile image82
    cashmereposted 9 years ago

    One of my hubs has been tried twice and blocked at both crawls...now what?

    1. profile image0
      Pacal Votanposted 9 years agoin reply to this

      You posted this in the Report a problem... forum, so hp will answer at some point. Let's wait and see. smile

  5. cashmere profile image82
    cashmereposted 9 years ago

    No answer yet...

  6. Len Cannon profile image88
    Len Cannonposted 9 years ago

    Huh.  I looked too and most of my top performers are blocked were listed as blocked as well.  I wonder if that has anything to do with my ads suddenly a lot of five cent clicks when I was getting fifty cents or more before.

  7. cashmere profile image82
    cashmereposted 9 years ago

    I have not noticed any drop in the price per click...but I do need these hubs unblocked

  8. Whitney05 profile image84
    Whitney05posted 9 years ago

    How do you check to see if they're blocked?

  9. frogdropping profile image84
    frogdroppingposted 9 years ago

    Whitney - just go to your adsense account. Click on the 'site diagnostics' tab. Anything blocked will show up in there.

    I have four pages. None (that I'm aware of) go against the TOS so I too am unsure as to why.

  10. Whitney05 profile image84
    Whitney05posted 9 years ago

    Gotcha.. Thanks.

  11. emievil profile image77
    emievilposted 9 years ago

    Just checked (thanks froggy for your reply to Whitney). Have seven of mine blocked due to the same reason. But my adsense earnings were still okay during the last crawl attempt.

  12. profile image0
    Pacal Votanposted 9 years ago

    We could probably figure out more if someone said which hub is blocked or by which line it is blocked. smile

    1. cashmere profile image82
      cashmereposted 9 years agoin reply to this

      I mentioned my blocked hubs in the thread starter post

      1. profile image0
        Pacal Votanposted 9 years agoin reply to this

        Oh, I'm sorry. Must have overlooked.
        Paul already said it was/is probably a glitch and should sort itself out.

  13. pauldeeds profile imageSTAFF
    pauldeedsposted 9 years ago

    Our robots.txt doesn't block the Google AdSense bot from any page (it's separate from the crawler for search, although they probably do share data).  I suspect these occasional messages are simply errors, but it's hard to tell without seeing the entire blocked URL.

  14. frogdropping profile image84
    frogdroppingposted 9 years ago

    Mine are - in no particular order -

    How To Wear White Jeans

    16 Weeks Pregnant

    Gifts For Teens - Boys

    Trumpett Socks

    You are welcome to the funky .txt if you want it smile

    Paul - thankyou.

    1. emievil profile image77
      emievilposted 9 years agoin reply to this

      Uhm, the URLs are a bit long. Will giving just the title (same as froggy) work as well?

  15. frogdropping profile image84
    frogdroppingposted 9 years ago

    http://hubpages.com/hub/My-very-First-Pet

    The above (as is) is at the top of the blocked list. I believe the hub is yours pacal?

    1. profile image0
      Pacal Votanposted 9 years agoin reply to this

      lolol but it's not a commercial hub anyway.

      the only way it was blocked by robots.txt is if the url had been changes, for instance if the hub had been flagged, also I found no noindex attribute in the source

      and how the heck do you see my hub in your analytics?

      1. frogdropping profile image84
        frogdroppingposted 9 years agoin reply to this

        No idea. It foxed me for a few minutes then (as I've read your hubs) I remembered whose it was.

        But it's there. At the top smile

        1. profile image0
          Pacal Votanposted 9 years agoin reply to this

          lol, then you know more about my hubs than me, because I don't even have GA set up for hubpages.

          no wonder though, because you are the frog

        2. Marisa Wright profile image96
          Marisa Wrightposted 9 years agoin reply to this

          Was this in Analytics or Adsense - or do you have them linked?

          I have someone else's Hub in my blocked list, too.  It threw me for a minute - then I realised he's one of my referrals, so my code must have been showing when Google crawled the Hub.

    2. ngureco profile image81
      ngurecoposted 9 years agoin reply to this

      How can you get that link without going into his adsense account?

  16. ngureco profile image81
    ngurecoposted 9 years ago

    Mine that are listed as blocked are still displaying my adsense id so no need to worry for now.

  17. frogdropping profile image84
    frogdroppingposted 9 years ago

    Yes. I'm a radar sneaky frog smile *nods*

    I will try a screen shot. You can see for yourself.

    1. profile image0
      Pacal Votanposted 9 years agoin reply to this

      I believe you. smile I think, I know why that is.

  18. frogdropping profile image84
    frogdroppingposted 9 years ago

    Ohhhhhh ok smile

    1. profile image0
      Pacal Votanposted 9 years agoin reply to this

      Why, did you really think I don't believe you? lol

      http://2.bp.blogspot.com/_drUMJ9HF-tQ/SIDvDRy2OrI/AAAAAAAAC44/myfHbfpwJ3Q/s400/Monster+Hug.jpg

  19. Fiction Teller profile image60
    Fiction Tellerposted 9 years ago

    I'm not a techie by any means, but could it possibly have to do with whether or not people access these pages from a location that is not one of your "Allowed sites?"

    I ask because some of my robot alerts start with:
    http:/ / translate. googleusercontent. com

    and, interestingly, translate.googleusercontent.com was not in my Allowed Sites but had been displaying my Google AdSense code in the last week.

    If you're worried, check back in your Site Diagnostics in a week, since I think they crawl no more than once a week.  When this has happened to me before, the blocked listing disappeared at the next crawl.

    1. pauldeeds profile imageSTAFF
      pauldeedsposted 9 years agoin reply to this

      Any time the HTML from your hub is served on another domain, such as through Google's translation service or a proxy, it will be that sites robot.txt that determines whether the Google MediaPartner's bot can crawl it.  There is nothing that can be done about those.

      If you are seeing just normal HubPages URL (ie. http://hubpages.com/hub/...) in that list, I have no good explanation for it (we aren't seeing any of those ourselves).  However, if you are seeing AdSense ads on the hub (in particular, well targeted AdSense ads), then I wouldn't worry too much about it.  It's probably just a glitch.

  20. Whitney05 profile image84
    Whitney05posted 9 years ago

    I have 4 pages of hubs on the list, to include hubs like:

    Cropping your dogs ears and docking tail
    miniature pets
    Bomberman land cheats
    Drag queen fashion
    Call of duty
    famous pit bulls
    rechargeable shaver models
    decorating christmas tree
    outdoor christmas decorations
    ww2 weapons
    mrsa staph infection
    k9 puppy gold
    knife sharpeners
    natural habitat of leopard gecko
    take apart an xbox
    gods of egyptian mythology
    hdmi dvd player
    rottweilers
    crysis
    pit bull movies
    home recycle bins
    protein shakes
    halo 3 armor (one of my most trafficked hub)
    advantages and disadvantages of homeschooling
    superworms
    kane and lynch
    60 inch plasma


    and a bunch from my referrels

  21. Whitney05 profile image84
    Whitney05posted 9 years ago

    So, what happens if you see the http://hubpages.com/hub/

    I see it on the top of my list.

    What if you're not seeing targeted ads? Some of mine that are the aboe list are not targed ads. On the native habitat for leopard geckos the ads include crazy white teeth, truth about belly fat, fix.op

  22. thranax profile image51
    thranaxposted 9 years ago

    Personally, I think this might be happening because of the server problem Hubpages had yesterday morning I think. Check the times it said it last tried to crawl the pages.

    ~thranax~

  23. cashmere profile image82
    cashmereposted 9 years ago

    Last time was Nov 5th

  24. Eileen Hughes profile image81
    Eileen Hughesposted 9 years ago

    Its the first time I have been into those diagnostics and I found 4 blocked

    They say  Robots.txt file:
    If you have a robots.txt file, the page requesting Google ads may be marked as 'disallow' in your robots.txt file. You can update your robots.txt file to give us access by following these instructions.
    If you do not have a robots.txt file, please create one in the root directory of your domain and then update it by following these instructions.
    If you don't know whether you have a robots.txt file, please consult your web provider or webhost and ask them to modify the robots.txt file on your domain to comply with these instructions.

    Do we or dont we have on in the root directory of oyr domain.
    Please help
    I know nothing about that.......

    1. profile image0
      Pacal Votanposted 9 years agoin reply to this

      A robots.txt is a must, because if a site doesn't have one Googlebot won't even bother crawling that site. Hp has one, of course.

      And look what I've found: http://leadlightphoto.blogspot.com/robots.txt

  25. Jerilee Wei profile image84
    Jerilee Weiposted 9 years ago

    Well, as a senior citizen I don't understand all this, and only found two but did learn some things I didn't know from this discussion.

  26. cashmere profile image82
    cashmereposted 9 years ago

    How many people still have blocked hubs showing?
    Now I have this in my blocked sites list
    http:/ / hubpages. com/ hub/
    Does that mean all my hubs are blocked?

    1. Glenn Stok profile image98
      Glenn Stokposted 9 years agoin reply to this

      cashmere, take a look back about 10 posts were Paul Deeds (Hubpages Staff) gave a good explanation.  I looked at the robots.txt file ... HubPages does not block any individual hubs at all.

      Two things can be happening that is confusing to see...

      (1) If you give referral links to people and they click on it, your affiliate code is placed in that other Hubbers ads. I think for 10% of the time, so you get 10% of their earnings for the referral. As I understand it, that 10% is part of HubPages 40%, so the other Hubber still gets his or her 60%. Anyway, when Google AdSense and Analytics sees your affiliate code in the other Hubber's Hub, that hub will appear in your report.  This is all normal and it is what gives you the credit for the referral.

      (2) The other thing that's confusing is that some web sites may serve your html (possibly in a frame) to show what you have written. If Googles see that the domain part of the URL is not one of your approved domains, then it will be rejected. This has nothing to do with anything that HubPages is doing.

      Hope that clarifies a few things for those I've been reading in this thread.

  27. cashmere profile image82
    cashmereposted 9 years ago

    Thanks Glenn.
    Now I get it. So it does not affect my hubs in any way. And nothing happens to the adsense either.

  28. Benz B profile image61
    Benz Bposted 9 years ago

    I'm having that same issue as Cashmere.  And I notice a dramaitic drop in my Adsense earnings since this started over the weekend.  I have had several consecutive days without even getting $0.01 for page impressions and I have hundreds.  There has to be something more to this problem.  I have 4 hubs all showing blocked even after a new crawl that added the 4th hub to the blocked list.  Here is what is showing on Adsense Diagnotics report:



    Blocked URL      Reason Blocked [?]     Last Crawl Attempt
    http:/ / hubpages. com/ hub/     Not Found     Nov 17, 2009
    The-Best-Cat-Flea-Control-Frontline-Advantage-Bio-Spot-Sentry? done     Not Found     Nov 17, 2009
    keys-to-becoming-a-successful-seller-on-ebay? done     Not Found     Nov 15, 2009
    top-tips-for-enjoying-a-football-game? done     Not Found     Nov 15, 2009
    ways-to-make-money-on-ebay? done     Not Found     Nov 12, 2009

  29. cashmere profile image82
    cashmereposted 9 years ago

    You know now that I have been monitoring it? The Adsense has dropped some what compared to the last couple of month's.

    1. cashmere profile image82
      cashmereposted 9 years agoin reply to this

      Its down to an all time low!

  30. Glenn Stok profile image98
    Glenn Stokposted 9 years ago

    I don't seem to be having the problems I read about in this thread. Everyone...try adding your URL's of all your hubs directly into your AdSense account. See if that helps and report your results here after a few days. That is what I'm doing. I am guessing that Google is looking for the URL with "?done" at the end, as "Benz B" reported. I don't know where Google is picking that up from. But if you put the URL's in yourself without anything extra, then Google should have no trouble finding it.

  31. cashmere profile image82
    cashmereposted 9 years ago

    I need to check for this "?done" bit. Thanks Glenn just going to check it out.

 
working

This website uses cookies

As a user in the EEA, your approval is needed on a few things. To provide a better website experience, hubpages.com uses cookies (and other similar technologies) and may collect, process, and share personal data. Please choose which areas of our service you consent to our doing so.

For more information on managing or withdrawing consents and how we handle data, visit our Privacy Policy at: https://hubpages.com/privacy-policy#gdpr

Show Details
Necessary
HubPages Device IDThis is used to identify particular browsers or devices when the access the service, and is used for security reasons.
LoginThis is necessary to sign in to the HubPages Service.
Google RecaptchaThis is used to prevent bots and spam. (Privacy Policy)
AkismetThis is used to detect comment spam. (Privacy Policy)
HubPages Google AnalyticsThis is used to provide data on traffic to our website, all personally identifyable data is anonymized. (Privacy Policy)
HubPages Traffic PixelThis is used to collect data on traffic to articles and other pages on our site. Unless you are signed in to a HubPages account, all personally identifiable information is anonymized.
Amazon Web ServicesThis is a cloud services platform that we used to host our service. (Privacy Policy)
CloudflareThis is a cloud CDN service that we use to efficiently deliver files required for our service to operate such as javascript, cascading style sheets, images, and videos. (Privacy Policy)
Google Hosted LibrariesJavascript software libraries such as jQuery are loaded at endpoints on the googleapis.com or gstatic.com domains, for performance and efficiency reasons. (Privacy Policy)
Features
Google Custom SearchThis is feature allows you to search the site. (Privacy Policy)
Google MapsSome articles have Google Maps embedded in them. (Privacy Policy)
Google ChartsThis is used to display charts and graphs on articles and the author center. (Privacy Policy)
Google AdSense Host APIThis service allows you to sign up for or associate a Google AdSense account with HubPages, so that you can earn money from ads on your articles. No data is shared unless you engage with this feature. (Privacy Policy)
Google YouTubeSome articles have YouTube videos embedded in them. (Privacy Policy)
VimeoSome articles have Vimeo videos embedded in them. (Privacy Policy)
PaypalThis is used for a registered author who enrolls in the HubPages Earnings program and requests to be paid via PayPal. No data is shared with Paypal unless you engage with this feature. (Privacy Policy)
Facebook LoginYou can use this to streamline signing up for, or signing in to your Hubpages account. No data is shared with Facebook unless you engage with this feature. (Privacy Policy)
MavenThis supports the Maven widget and search functionality. (Privacy Policy)
Marketing
Google AdSenseThis is an ad network. (Privacy Policy)
Google DoubleClickGoogle provides ad serving technology and runs an ad network. (Privacy Policy)
Index ExchangeThis is an ad network. (Privacy Policy)
SovrnThis is an ad network. (Privacy Policy)
Facebook AdsThis is an ad network. (Privacy Policy)
Amazon Unified Ad MarketplaceThis is an ad network. (Privacy Policy)
AppNexusThis is an ad network. (Privacy Policy)
OpenxThis is an ad network. (Privacy Policy)
Rubicon ProjectThis is an ad network. (Privacy Policy)
TripleLiftThis is an ad network. (Privacy Policy)
Say MediaWe partner with Say Media to deliver ad campaigns on our sites. (Privacy Policy)
Remarketing PixelsWe may use remarketing pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to advertise the HubPages Service to people that have visited our sites.
Conversion Tracking PixelsWe may use conversion tracking pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to identify when an advertisement has successfully resulted in the desired action, such as signing up for the HubPages Service or publishing an article on the HubPages Service.
Statistics
Author Google AnalyticsThis is used to provide traffic data and reports to the authors of articles on the HubPages Service. (Privacy Policy)
ComscoreComScore is a media measurement and analytics company providing marketing data and analytics to enterprises, media and advertising agencies, and publishers. Non-consent will result in ComScore only processing obfuscated personal data. (Privacy Policy)
Amazon Tracking PixelSome articles display amazon products as part of the Amazon Affiliate program, this pixel provides traffic statistics for those products (Privacy Policy)