In my site diagnostics in Google adsense these hubs of mine are being shown as blocked. Reason blocked is Robots.txt File.
Can you tell me how I can unblock my following hubs?
My favourite Tarot layouts,
How to teach English
Multilevel marketing - the good the bad and ugly.
This has happened since Nov 5, but these hubs have been published months ago.
What happened and how can I unblock them?
Robots.txt is on hubpages server. It's where hubstaff specifies which pages search robots should not scan. For some reason they chose to hide these hubs of yours from search engines. You have no access to robots.txt so you can do nothing about it.
Okay, but why would they suddenly choose to block my hubs now? Three months after publication?
I never even looked at site diagnostics before! I just checked and four of my Hubs show as blocked, too.
The Help screen suggests checking again later, as it may just be that the server was down when they tried to crawl the site. Let's hope that's the reason.
One of my hubs has been tried twice and blocked at both crawls...now what?
Huh. I looked too and most of my top performers are blocked were listed as blocked as well. I wonder if that has anything to do with my ads suddenly a lot of five cent clicks when I was getting fifty cents or more before.
I have not noticed any drop in the price per click...but I do need these hubs unblocked
Whitney - just go to your adsense account. Click on the 'site diagnostics' tab. Anything blocked will show up in there.
I have four pages. None (that I'm aware of) go against the TOS so I too am unsure as to why.
Just checked (thanks froggy for your reply to Whitney). Have seven of mine blocked due to the same reason. But my adsense earnings were still okay during the last crawl attempt.
We could probably figure out more if someone said which hub is blocked or by which line it is blocked.
Our robots.txt doesn't block the Google AdSense bot from any page (it's separate from the crawler for search, although they probably do share data). I suspect these occasional messages are simply errors, but it's hard to tell without seeing the entire blocked URL.
Mine are - in no particular order -
How To Wear White Jeans
16 Weeks Pregnant
Gifts For Teens - Boys
You are welcome to the funky .txt if you want it
Paul - thankyou.
The above (as is) is at the top of the blocked list. I believe the hub is yours pacal?
lolol but it's not a commercial hub anyway.
the only way it was blocked by robots.txt is if the url had been changes, for instance if the hub had been flagged, also I found no noindex attribute in the source
and how the heck do you see my hub in your analytics?
No idea. It foxed me for a few minutes then (as I've read your hubs) I remembered whose it was.
But it's there. At the top
lol, then you know more about my hubs than me, because I don't even have GA set up for hubpages.
no wonder though, because you are the frog
Was this in Analytics or Adsense - or do you have them linked?
I have someone else's Hub in my blocked list, too. It threw me for a minute - then I realised he's one of my referrals, so my code must have been showing when Google crawled the Hub.
How can you get that link without going into his adsense account?
Mine that are listed as blocked are still displaying my adsense id so no need to worry for now.
Yes. I'm a radar sneaky frog *nods*
I will try a screen shot. You can see for yourself.
I'm not a techie by any means, but could it possibly have to do with whether or not people access these pages from a location that is not one of your "Allowed sites?"
I ask because some of my robot alerts start with:
http:/ / translate. googleusercontent. com
and, interestingly, translate.googleusercontent.com was not in my Allowed Sites but had been displaying my Google AdSense code in the last week.
If you're worried, check back in your Site Diagnostics in a week, since I think they crawl no more than once a week. When this has happened to me before, the blocked listing disappeared at the next crawl.
Any time the HTML from your hub is served on another domain, such as through Google's translation service or a proxy, it will be that sites robot.txt that determines whether the Google MediaPartner's bot can crawl it. There is nothing that can be done about those.
If you are seeing just normal HubPages URL (ie. http://hubpages.com/hub/...) in that list, I have no good explanation for it (we aren't seeing any of those ourselves). However, if you are seeing AdSense ads on the hub (in particular, well targeted AdSense ads), then I wouldn't worry too much about it. It's probably just a glitch.
I have 4 pages of hubs on the list, to include hubs like:
Cropping your dogs ears and docking tail
Bomberman land cheats
Drag queen fashion
Call of duty
famous pit bulls
rechargeable shaver models
decorating christmas tree
outdoor christmas decorations
mrsa staph infection
k9 puppy gold
natural habitat of leopard gecko
take apart an xbox
gods of egyptian mythology
hdmi dvd player
pit bull movies
home recycle bins
halo 3 armor (one of my most trafficked hub)
advantages and disadvantages of homeschooling
kane and lynch
60 inch plasma
and a bunch from my referrels
So, what happens if you see the http://hubpages.com/hub/
I see it on the top of my list.
What if you're not seeing targeted ads? Some of mine that are the aboe list are not targed ads. On the native habitat for leopard geckos the ads include crazy white teeth, truth about belly fat, fix.op
Personally, I think this might be happening because of the server problem Hubpages had yesterday morning I think. Check the times it said it last tried to crawl the pages.
Its the first time I have been into those diagnostics and I found 4 blocked
They say Robots.txt file:
If you have a robots.txt file, the page requesting Google ads may be marked as 'disallow' in your robots.txt file. You can update your robots.txt file to give us access by following these instructions.
If you do not have a robots.txt file, please create one in the root directory of your domain and then update it by following these instructions.
If you don't know whether you have a robots.txt file, please consult your web provider or webhost and ask them to modify the robots.txt file on your domain to comply with these instructions.
Do we or dont we have on in the root directory of oyr domain.
I know nothing about that.......
Well, as a senior citizen I don't understand all this, and only found two but did learn some things I didn't know from this discussion.
How many people still have blocked hubs showing?
Now I have this in my blocked sites list
http:/ / hubpages. com/ hub/
Does that mean all my hubs are blocked?
cashmere, take a look back about 10 posts were Paul Deeds (Hubpages Staff) gave a good explanation. I looked at the robots.txt file ... HubPages does not block any individual hubs at all.
Two things can be happening that is confusing to see...
(1) If you give referral links to people and they click on it, your affiliate code is placed in that other Hubbers ads. I think for 10% of the time, so you get 10% of their earnings for the referral. As I understand it, that 10% is part of HubPages 40%, so the other Hubber still gets his or her 60%. Anyway, when Google AdSense and Analytics sees your affiliate code in the other Hubber's Hub, that hub will appear in your report. This is all normal and it is what gives you the credit for the referral.
(2) The other thing that's confusing is that some web sites may serve your html (possibly in a frame) to show what you have written. If Googles see that the domain part of the URL is not one of your approved domains, then it will be rejected. This has nothing to do with anything that HubPages is doing.
Hope that clarifies a few things for those I've been reading in this thread.
Now I get it. So it does not affect my hubs in any way. And nothing happens to the adsense either.
I'm having that same issue as Cashmere. And I notice a dramaitic drop in my Adsense earnings since this started over the weekend. I have had several consecutive days without even getting $0.01 for page impressions and I have hundreds. There has to be something more to this problem. I have 4 hubs all showing blocked even after a new crawl that added the 4th hub to the blocked list. Here is what is showing on Adsense Diagnotics report:
Blocked URL Reason Blocked [?] Last Crawl Attempt
http:/ / hubpages. com/ hub/ Not Found Nov 17, 2009
The-Best-Cat-Flea-Control-Frontline-Advantage-Bio-Spot-Sentry? done Not Found Nov 17, 2009
keys-to-becoming-a-successful-seller-on-ebay? done Not Found Nov 15, 2009
top-tips-for-enjoying-a-football-game? done Not Found Nov 15, 2009
ways-to-make-money-on-ebay? done Not Found Nov 12, 2009
You know now that I have been monitoring it? The Adsense has dropped some what compared to the last couple of month's.
I don't seem to be having the problems I read about in this thread. Everyone...try adding your URL's of all your hubs directly into your AdSense account. See if that helps and report your results here after a few days. That is what I'm doing. I am guessing that Google is looking for the URL with "?done" at the end, as "Benz B" reported. I don't know where Google is picking that up from. But if you put the URL's in yourself without anything extra, then Google should have no trouble finding it.
I need to check for this "?done" bit. Thanks Glenn just going to check it out.
by Quilligrapher8 years ago
Google AdSense Site Diagnostics reports that a Robots.txt file is blocking their crawler from accessing one of my hubs. Can anyone suggest how I can resolve this issue?
by Mike Bouska8 years ago
When I logged into Adsense today, it gave me an Analytics diagnostic warning for hubpages, saying that they could not crawl because of a Robots.txt error.Anyone else getting this?
by Brinafr3sh5 years ago
Have any of your Hubs been blocked due to robots.txt file?Did you ever go to one of your Hubs url's and it said "robots.txt file," instead of it showing your hubsite?
by Vibhavari8 years ago
Hi I was checking my adsense account and in the reports section a number of my URLs appeared as blocked URL and the reason stated was the Robots.txt file. What is a Robots.txt file?Does a blocked URL mean that the...
by kimkay4 months ago
I just received the below message from Adsense.Publisher ID missing from ads.txt filesYou need to add your publisher ID to the following ads.txt files: hubpages.com/ads.txtThis will prevent a potentially...
by Tammy6 years ago
I tried to asked this question before, but it didn't show up. I logged into Google Webmaster Tools and it said restricted by robots.txt. What is that and how do I fix it? It looks something like this:Show URLs:...
Copyright © 2018 HubPages Inc. and respective owners.
Other product and company names shown may be trademarks of their respective owners.
HubPages® is a registered Service Mark of HubPages, Inc.
HubPages and Hubbers (authors) may earn revenue on this page based on affiliate relationships and advertisements with partners including Amazon, Google, and others.