jump to last post 1-31 of 31 discussions (47 posts)

Google Adsense blocked due to Robots.txt File

  1. cashmere profile image82
    cashmereposted 7 years ago

    In my site diagnostics in Google adsense these hubs of mine are being shown as blocked. Reason blocked is Robots.txt File.
    Can you tell me how I can unblock my following hubs?

    My favourite Tarot layouts,
    How to teach English
    and
    Multilevel marketing - the good the bad and ugly.

    This has happened since Nov 5, but these hubs have been published months ago.
    What happened and how can I unblock them?

    1. 0
      Pacal Votanposted 7 years ago in reply to this

      Robots.txt is on hubpages server. It's where hubstaff specifies which pages search robots should not scan. For some reason they chose to hide these hubs of yours from search engines. You have no access to robots.txt so you can do nothing about it.

  2. cashmere profile image82
    cashmereposted 7 years ago

    Okay, but why would they suddenly choose to block my hubs now? Three months after publication?

  3. Marisa Wright profile image93
    Marisa Wrightposted 7 years ago

    I never even looked at site diagnostics before!  I just checked and four of my Hubs show as blocked, too.

    The Help screen suggests checking again later, as it may just be that the server was down when they tried to crawl the site.  Let's hope that's the reason.

  4. cashmere profile image82
    cashmereposted 7 years ago

    One of my hubs has been tried twice and blocked at both crawls...now what?

    1. 0
      Pacal Votanposted 7 years ago in reply to this

      You posted this in the Report a problem... forum, so hp will answer at some point. Let's wait and see. smile

  5. cashmere profile image82
    cashmereposted 7 years ago

    No answer yet...

  6. Len Cannon profile image87
    Len Cannonposted 7 years ago

    Huh.  I looked too and most of my top performers are blocked were listed as blocked as well.  I wonder if that has anything to do with my ads suddenly a lot of five cent clicks when I was getting fifty cents or more before.

  7. cashmere profile image82
    cashmereposted 7 years ago

    I have not noticed any drop in the price per click...but I do need these hubs unblocked

  8. Whitney05 profile image63
    Whitney05posted 7 years ago

    How do you check to see if they're blocked?

  9. frogdropping profile image85
    frogdroppingposted 7 years ago

    Whitney - just go to your adsense account. Click on the 'site diagnostics' tab. Anything blocked will show up in there.

    I have four pages. None (that I'm aware of) go against the TOS so I too am unsure as to why.

  10. Whitney05 profile image63
    Whitney05posted 7 years ago

    Gotcha.. Thanks.

  11. emievil profile image84
    emievilposted 7 years ago

    Just checked (thanks froggy for your reply to Whitney). Have seven of mine blocked due to the same reason. But my adsense earnings were still okay during the last crawl attempt.

  12. 0
    Pacal Votanposted 7 years ago

    We could probably figure out more if someone said which hub is blocked or by which line it is blocked. smile

    1. cashmere profile image82
      cashmereposted 7 years ago in reply to this

      I mentioned my blocked hubs in the thread starter post

      1. 0
        Pacal Votanposted 7 years ago in reply to this

        Oh, I'm sorry. Must have overlooked.
        Paul already said it was/is probably a glitch and should sort itself out.

  13. pauldeeds profile image
    62
    pauldeedsposted 7 years ago

    Our robots.txt doesn't block the Google AdSense bot from any page (it's separate from the crawler for search, although they probably do share data).  I suspect these occasional messages are simply errors, but it's hard to tell without seeing the entire blocked URL.

  14. frogdropping profile image85
    frogdroppingposted 7 years ago

    Mine are - in no particular order -

    How To Wear White Jeans

    16 Weeks Pregnant

    Gifts For Teens - Boys

    Trumpett Socks

    You are welcome to the funky .txt if you want it smile

    Paul - thankyou.

    1. emievil profile image84
      emievilposted 7 years ago in reply to this

      Uhm, the URLs are a bit long. Will giving just the title (same as froggy) work as well?

  15. frogdropping profile image85
    frogdroppingposted 7 years ago

    http://hubpages.com/hub/My-very-First-Pet

    The above (as is) is at the top of the blocked list. I believe the hub is yours pacal?

    1. 0
      Pacal Votanposted 7 years ago in reply to this

      lolol but it's not a commercial hub anyway.

      the only way it was blocked by robots.txt is if the url had been changes, for instance if the hub had been flagged, also I found no noindex attribute in the source

      and how the heck do you see my hub in your analytics?

      1. frogdropping profile image85
        frogdroppingposted 7 years ago in reply to this

        No idea. It foxed me for a few minutes then (as I've read your hubs) I remembered whose it was.

        But it's there. At the top smile

        1. 0
          Pacal Votanposted 7 years ago in reply to this

          lol, then you know more about my hubs than me, because I don't even have GA set up for hubpages.

          no wonder though, because you are the frog

        2. Marisa Wright profile image93
          Marisa Wrightposted 7 years ago in reply to this

          Was this in Analytics or Adsense - or do you have them linked?

          I have someone else's Hub in my blocked list, too.  It threw me for a minute - then I realised he's one of my referrals, so my code must have been showing when Google crawled the Hub.

    2. ngureco profile image86
      ngurecoposted 7 years ago in reply to this

      How can you get that link without going into his adsense account?

  16. ngureco profile image86
    ngurecoposted 7 years ago

    Mine that are listed as blocked are still displaying my adsense id so no need to worry for now.

  17. frogdropping profile image85
    frogdroppingposted 7 years ago

    Yes. I'm a radar sneaky frog smile *nods*

    I will try a screen shot. You can see for yourself.

    1. 0
      Pacal Votanposted 7 years ago in reply to this

      I believe you. smile I think, I know why that is.

  18. frogdropping profile image85
    frogdroppingposted 7 years ago

    Ohhhhhh ok smile

    1. 0
      Pacal Votanposted 7 years ago in reply to this

      Why, did you really think I don't believe you? lol

      http://2.bp.blogspot.com/_drUMJ9HF-tQ/SIDvDRy2OrI/AAAAAAAAC44/myfHbfpwJ3Q/s400/Monster+Hug.jpg

  19. Fiction Teller profile image60
    Fiction Tellerposted 7 years ago

    I'm not a techie by any means, but could it possibly have to do with whether or not people access these pages from a location that is not one of your "Allowed sites?"

    I ask because some of my robot alerts start with:
    http:/ / translate. googleusercontent. com

    and, interestingly, translate.googleusercontent.com was not in my Allowed Sites but had been displaying my Google AdSense code in the last week.

    If you're worried, check back in your Site Diagnostics in a week, since I think they crawl no more than once a week.  When this has happened to me before, the blocked listing disappeared at the next crawl.

    1. pauldeeds profile image
      62
      pauldeedsposted 7 years ago in reply to this

      Any time the HTML from your hub is served on another domain, such as through Google's translation service or a proxy, it will be that sites robot.txt that determines whether the Google MediaPartner's bot can crawl it.  There is nothing that can be done about those.

      If you are seeing just normal HubPages URL (ie. http://hubpages.com/hub/...) in that list, I have no good explanation for it (we aren't seeing any of those ourselves).  However, if you are seeing AdSense ads on the hub (in particular, well targeted AdSense ads), then I wouldn't worry too much about it.  It's probably just a glitch.

  20. Whitney05 profile image63
    Whitney05posted 7 years ago

    I have 4 pages of hubs on the list, to include hubs like:

    Cropping your dogs ears and docking tail
    miniature pets
    Bomberman land cheats
    Drag queen fashion
    Call of duty
    famous pit bulls
    rechargeable shaver models
    decorating christmas tree
    outdoor christmas decorations
    ww2 weapons
    mrsa staph infection
    k9 puppy gold
    knife sharpeners
    natural habitat of leopard gecko
    take apart an xbox
    gods of egyptian mythology
    hdmi dvd player
    rottweilers
    crysis
    pit bull movies
    home recycle bins
    protein shakes
    halo 3 armor (one of my most trafficked hub)
    advantages and disadvantages of homeschooling
    superworms
    kane and lynch
    60 inch plasma


    and a bunch from my referrels

  21. Whitney05 profile image63
    Whitney05posted 7 years ago

    So, what happens if you see the http://hubpages.com/hub/

    I see it on the top of my list.

    What if you're not seeing targeted ads? Some of mine that are the aboe list are not targed ads. On the native habitat for leopard geckos the ads include crazy white teeth, truth about belly fat, fix.op

  22. thranax profile image59
    thranaxposted 7 years ago

    Personally, I think this might be happening because of the server problem Hubpages had yesterday morning I think. Check the times it said it last tried to crawl the pages.

    ~thranax~

  23. cashmere profile image82
    cashmereposted 7 years ago

    Last time was Nov 5th

  24. Eileen Hughes profile image82
    Eileen Hughesposted 7 years ago

    Its the first time I have been into those diagnostics and I found 4 blocked

    They say  Robots.txt file:
    If you have a robots.txt file, the page requesting Google ads may be marked as 'disallow' in your robots.txt file. You can update your robots.txt file to give us access by following these instructions.
    If you do not have a robots.txt file, please create one in the root directory of your domain and then update it by following these instructions.
    If you don't know whether you have a robots.txt file, please consult your web provider or webhost and ask them to modify the robots.txt file on your domain to comply with these instructions.

    Do we or dont we have on in the root directory of oyr domain.
    Please help
    I know nothing about that.......

    1. 0
      Pacal Votanposted 7 years ago in reply to this

      A robots.txt is a must, because if a site doesn't have one Googlebot won't even bother crawling that site. Hp has one, of course.

      And look what I've found: http://leadlightphoto.blogspot.com/robots.txt

  25. Jerilee Wei profile image93
    Jerilee Weiposted 7 years ago

    Well, as a senior citizen I don't understand all this, and only found two but did learn some things I didn't know from this discussion.

  26. cashmere profile image82
    cashmereposted 7 years ago

    How many people still have blocked hubs showing?
    Now I have this in my blocked sites list
    http:/ / hubpages. com/ hub/
    Does that mean all my hubs are blocked?

    1. Glenn Stok profile image98
      Glenn Stokposted 7 years ago in reply to this

      cashmere, take a look back about 10 posts were Paul Deeds (Hubpages Staff) gave a good explanation.  I looked at the robots.txt file ... HubPages does not block any individual hubs at all.

      Two things can be happening that is confusing to see...

      (1) If you give referral links to people and they click on it, your affiliate code is placed in that other Hubbers ads. I think for 10% of the time, so you get 10% of their earnings for the referral. As I understand it, that 10% is part of HubPages 40%, so the other Hubber still gets his or her 60%. Anyway, when Google AdSense and Analytics sees your affiliate code in the other Hubber's Hub, that hub will appear in your report.  This is all normal and it is what gives you the credit for the referral.

      (2) The other thing that's confusing is that some web sites may serve your html (possibly in a frame) to show what you have written. If Googles see that the domain part of the URL is not one of your approved domains, then it will be rejected. This has nothing to do with anything that HubPages is doing.

      Hope that clarifies a few things for those I've been reading in this thread.

  27. cashmere profile image82
    cashmereposted 7 years ago

    Thanks Glenn.
    Now I get it. So it does not affect my hubs in any way. And nothing happens to the adsense either.

  28. Benz B profile image60
    Benz Bposted 7 years ago

    I'm having that same issue as Cashmere.  And I notice a dramaitic drop in my Adsense earnings since this started over the weekend.  I have had several consecutive days without even getting $0.01 for page impressions and I have hundreds.  There has to be something more to this problem.  I have 4 hubs all showing blocked even after a new crawl that added the 4th hub to the blocked list.  Here is what is showing on Adsense Diagnotics report:



    Blocked URL      Reason Blocked [?]     Last Crawl Attempt
    http:/ / hubpages. com/ hub/     Not Found     Nov 17, 2009
    The-Best-Cat-Flea-Control-Frontline-Advantage-Bio-Spot-Sentry? done     Not Found     Nov 17, 2009
    keys-to-becoming-a-successful-seller-on-ebay? done     Not Found     Nov 15, 2009
    top-tips-for-enjoying-a-football-game? done     Not Found     Nov 15, 2009
    ways-to-make-money-on-ebay? done     Not Found     Nov 12, 2009

  29. cashmere profile image82
    cashmereposted 7 years ago

    You know now that I have been monitoring it? The Adsense has dropped some what compared to the last couple of month's.

    1. cashmere profile image82
      cashmereposted 7 years ago in reply to this

      Its down to an all time low!

  30. Glenn Stok profile image98
    Glenn Stokposted 7 years ago

    I don't seem to be having the problems I read about in this thread. Everyone...try adding your URL's of all your hubs directly into your AdSense account. See if that helps and report your results here after a few days. That is what I'm doing. I am guessing that Google is looking for the URL with "?done" at the end, as "Benz B" reported. I don't know where Google is picking that up from. But if you put the URL's in yourself without anything extra, then Google should have no trouble finding it.

  31. cashmere profile image82
    cashmereposted 7 years ago

    I need to check for this "?done" bit. Thanks Glenn just going to check it out.

 
working