In my site diagnostics in Google adsense these hubs of mine are being shown as blocked. Reason blocked is Robots.txt File.
Can you tell me how I can unblock my following hubs?
My favourite Tarot layouts,
How to teach English
and
Multilevel marketing - the good the bad and ugly.
This has happened since Nov 5, but these hubs have been published months ago.
What happened and how can I unblock them?
Robots.txt is on hubpages server. It's where hubstaff specifies which pages search robots should not scan. For some reason they chose to hide these hubs of yours from search engines. You have no access to robots.txt so you can do nothing about it.
Okay, but why would they suddenly choose to block my hubs now? Three months after publication?
I never even looked at site diagnostics before! I just checked and four of my Hubs show as blocked, too.
The Help screen suggests checking again later, as it may just be that the server was down when they tried to crawl the site. Let's hope that's the reason.
One of my hubs has been tried twice and blocked at both crawls...now what?
Huh. I looked too and most of my top performers are blocked were listed as blocked as well. I wonder if that has anything to do with my ads suddenly a lot of five cent clicks when I was getting fifty cents or more before.
I have not noticed any drop in the price per click...but I do need these hubs unblocked
Whitney - just go to your adsense account. Click on the 'site diagnostics' tab. Anything blocked will show up in there.
I have four pages. None (that I'm aware of) go against the TOS so I too am unsure as to why.
Just checked (thanks froggy for your reply to Whitney). Have seven of mine blocked due to the same reason. But my adsense earnings were still okay during the last crawl attempt.
We could probably figure out more if someone said which hub is blocked or by which line it is blocked.
Our robots.txt doesn't block the Google AdSense bot from any page (it's separate from the crawler for search, although they probably do share data). I suspect these occasional messages are simply errors, but it's hard to tell without seeing the entire blocked URL.
Mine are - in no particular order -
How To Wear White Jeans
16 Weeks Pregnant
Gifts For Teens - Boys
Trumpett Socks
You are welcome to the funky .txt if you want it
Paul - thankyou.
http://hubpages.com/hub/My-very-First-Pet
The above (as is) is at the top of the blocked list. I believe the hub is yours pacal?
lolol but it's not a commercial hub anyway.
the only way it was blocked by robots.txt is if the url had been changes, for instance if the hub had been flagged, also I found no noindex attribute in the source
and how the heck do you see my hub in your analytics?
No idea. It foxed me for a few minutes then (as I've read your hubs) I remembered whose it was.
But it's there. At the top
lol, then you know more about my hubs than me, because I don't even have GA set up for hubpages.
no wonder though, because you are the frog
Was this in Analytics or Adsense - or do you have them linked?
I have someone else's Hub in my blocked list, too. It threw me for a minute - then I realised he's one of my referrals, so my code must have been showing when Google crawled the Hub.
How can you get that link without going into his adsense account?
Mine that are listed as blocked are still displaying my adsense id so no need to worry for now.
Yes. I'm a radar sneaky frog *nods*
I will try a screen shot. You can see for yourself.
I'm not a techie by any means, but could it possibly have to do with whether or not people access these pages from a location that is not one of your "Allowed sites?"
I ask because some of my robot alerts start with:
http:/ / translate. googleusercontent. com
and, interestingly, translate.googleusercontent.com was not in my Allowed Sites but had been displaying my Google AdSense code in the last week.
If you're worried, check back in your Site Diagnostics in a week, since I think they crawl no more than once a week. When this has happened to me before, the blocked listing disappeared at the next crawl.
Any time the HTML from your hub is served on another domain, such as through Google's translation service or a proxy, it will be that sites robot.txt that determines whether the Google MediaPartner's bot can crawl it. There is nothing that can be done about those.
If you are seeing just normal HubPages URL (ie. http://hubpages.com/hub/...) in that list, I have no good explanation for it (we aren't seeing any of those ourselves). However, if you are seeing AdSense ads on the hub (in particular, well targeted AdSense ads), then I wouldn't worry too much about it. It's probably just a glitch.
I have 4 pages of hubs on the list, to include hubs like:
Cropping your dogs ears and docking tail
miniature pets
Bomberman land cheats
Drag queen fashion
Call of duty
famous pit bulls
rechargeable shaver models
decorating christmas tree
outdoor christmas decorations
ww2 weapons
mrsa staph infection
k9 puppy gold
knife sharpeners
natural habitat of leopard gecko
take apart an xbox
gods of egyptian mythology
hdmi dvd player
rottweilers
crysis
pit bull movies
home recycle bins
protein shakes
halo 3 armor (one of my most trafficked hub)
advantages and disadvantages of homeschooling
superworms
kane and lynch
60 inch plasma
and a bunch from my referrels
So, what happens if you see the http://hubpages.com/hub/
I see it on the top of my list.
What if you're not seeing targeted ads? Some of mine that are the aboe list are not targed ads. On the native habitat for leopard geckos the ads include crazy white teeth, truth about belly fat, fix.op
Personally, I think this might be happening because of the server problem Hubpages had yesterday morning I think. Check the times it said it last tried to crawl the pages.
~thranax~
Its the first time I have been into those diagnostics and I found 4 blocked
They say Robots.txt file:
If you have a robots.txt file, the page requesting Google ads may be marked as 'disallow' in your robots.txt file. You can update your robots.txt file to give us access by following these instructions.
If you do not have a robots.txt file, please create one in the root directory of your domain and then update it by following these instructions.
If you don't know whether you have a robots.txt file, please consult your web provider or webhost and ask them to modify the robots.txt file on your domain to comply with these instructions.
Do we or dont we have on in the root directory of oyr domain.
Please help
I know nothing about that.......
A robots.txt is a must, because if a site doesn't have one Googlebot won't even bother crawling that site. Hp has one, of course.
And look what I've found: http://leadlightphoto.blogspot.com/robots.txt
Well, as a senior citizen I don't understand all this, and only found two but did learn some things I didn't know from this discussion.
How many people still have blocked hubs showing?
Now I have this in my blocked sites list
http:/ / hubpages. com/ hub/
Does that mean all my hubs are blocked?
cashmere, take a look back about 10 posts were Paul Deeds (Hubpages Staff) gave a good explanation. I looked at the robots.txt file ... HubPages does not block any individual hubs at all.
Two things can be happening that is confusing to see...
(1) If you give referral links to people and they click on it, your affiliate code is placed in that other Hubbers ads. I think for 10% of the time, so you get 10% of their earnings for the referral. As I understand it, that 10% is part of HubPages 40%, so the other Hubber still gets his or her 60%. Anyway, when Google AdSense and Analytics sees your affiliate code in the other Hubber's Hub, that hub will appear in your report. This is all normal and it is what gives you the credit for the referral.
(2) The other thing that's confusing is that some web sites may serve your html (possibly in a frame) to show what you have written. If Googles see that the domain part of the URL is not one of your approved domains, then it will be rejected. This has nothing to do with anything that HubPages is doing.
Hope that clarifies a few things for those I've been reading in this thread.
Thanks Glenn.
Now I get it. So it does not affect my hubs in any way. And nothing happens to the adsense either.
I'm having that same issue as Cashmere. And I notice a dramaitic drop in my Adsense earnings since this started over the weekend. I have had several consecutive days without even getting $0.01 for page impressions and I have hundreds. There has to be something more to this problem. I have 4 hubs all showing blocked even after a new crawl that added the 4th hub to the blocked list. Here is what is showing on Adsense Diagnotics report:
Blocked URL Reason Blocked [?] Last Crawl Attempt
http:/ / hubpages. com/ hub/ Not Found Nov 17, 2009
The-Best-Cat-Flea-Control-Frontline-Advantage-Bio-Spot-Sentry? done Not Found Nov 17, 2009
keys-to-becoming-a-successful-seller-on-ebay? done Not Found Nov 15, 2009
top-tips-for-enjoying-a-football-game? done Not Found Nov 15, 2009
ways-to-make-money-on-ebay? done Not Found Nov 12, 2009
You know now that I have been monitoring it? The Adsense has dropped some what compared to the last couple of month's.
I don't seem to be having the problems I read about in this thread. Everyone...try adding your URL's of all your hubs directly into your AdSense account. See if that helps and report your results here after a few days. That is what I'm doing. I am guessing that Google is looking for the URL with "?done" at the end, as "Benz B" reported. I don't know where Google is picking that up from. But if you put the URL's in yourself without anything extra, then Google should have no trouble finding it.
I need to check for this "?done" bit. Thanks Glenn just going to check it out.
by geekMachine 10 years ago
Just after being moved from Squidoo, I tried to apply for an AdSense account, which failed for this reason:"Insufficient content: To be approved for AdSense and show relevant ads on your site, your pages need to have enough text on them for our specialists to review and for our crawler to be...
by Alex L. Soriano 11 years ago
Why Google Adsense did not approve my account? I already have 10 unique articles here. I have no choice but to remove all of my articles from here. Hello Alexander L Soriano,Thank you for your interest in Google AdSense. After reviewing yourapplication, our specialists have found that it does not...
by Quilligrapher 15 years ago
Google AdSense Site Diagnostics reports that a Robots.txt file is blocking their crawler from accessing one of my hubs. Can anyone suggest how I can resolve this issue?
by Ananta65 14 years ago
Well, that’s nice, very nice. I haven’t got a clue of what I did wrong. And after filling in the online appeal form they told me they had reviewed my account information, taken my feedback into account, but still had concluded that my AdSense account has posed a significant risk. And I have no idea...
by Nu Vew 9 years ago
"During our review of your website, we found your site difficult to navigate. Potential navigation issues include: redirects, pages behind a login or restricted access, broken links, excessive pop-ups, dialers, and pages under construction or not yet launched."It makes no sense. I had...
by Jasmeet Kaur 11 years ago
Google adsense approval is a major issue that is faced by several people on hubpages. After waiting for 6 months, I got preliminary approval for google adsense (3 days passed). I was very happy to see this that finally I succed, but today I got mail stating that my application is rejected. I can't...
Copyright © 2025 The Arena Media Brands, LLC and respective content providers on this website. HubPages® is a registered trademark of The Arena Platform, Inc. Other product and company names shown may be trademarks of their respective owners. The Arena Media Brands, LLC and respective content providers to this website may receive compensation for some links to products and services on this website.
Copyright © 2025 Maven Media Brands, LLC and respective owners.
As a user in the EEA, your approval is needed on a few things. To provide a better website experience, hubpages.com uses cookies (and other similar technologies) and may collect, process, and share personal data. Please choose which areas of our service you consent to our doing so.
For more information on managing or withdrawing consents and how we handle data, visit our Privacy Policy at: https://corp.maven.io/privacy-policy
Show DetailsNecessary | |
---|---|
HubPages Device ID | This is used to identify particular browsers or devices when the access the service, and is used for security reasons. |
Login | This is necessary to sign in to the HubPages Service. |
Google Recaptcha | This is used to prevent bots and spam. (Privacy Policy) |
Akismet | This is used to detect comment spam. (Privacy Policy) |
HubPages Google Analytics | This is used to provide data on traffic to our website, all personally identifyable data is anonymized. (Privacy Policy) |
HubPages Traffic Pixel | This is used to collect data on traffic to articles and other pages on our site. Unless you are signed in to a HubPages account, all personally identifiable information is anonymized. |
Amazon Web Services | This is a cloud services platform that we used to host our service. (Privacy Policy) |
Cloudflare | This is a cloud CDN service that we use to efficiently deliver files required for our service to operate such as javascript, cascading style sheets, images, and videos. (Privacy Policy) |
Google Hosted Libraries | Javascript software libraries such as jQuery are loaded at endpoints on the googleapis.com or gstatic.com domains, for performance and efficiency reasons. (Privacy Policy) |
Features | |
---|---|
Google Custom Search | This is feature allows you to search the site. (Privacy Policy) |
Google Maps | Some articles have Google Maps embedded in them. (Privacy Policy) |
Google Charts | This is used to display charts and graphs on articles and the author center. (Privacy Policy) |
Google AdSense Host API | This service allows you to sign up for or associate a Google AdSense account with HubPages, so that you can earn money from ads on your articles. No data is shared unless you engage with this feature. (Privacy Policy) |
Google YouTube | Some articles have YouTube videos embedded in them. (Privacy Policy) |
Vimeo | Some articles have Vimeo videos embedded in them. (Privacy Policy) |
Paypal | This is used for a registered author who enrolls in the HubPages Earnings program and requests to be paid via PayPal. No data is shared with Paypal unless you engage with this feature. (Privacy Policy) |
Facebook Login | You can use this to streamline signing up for, or signing in to your Hubpages account. No data is shared with Facebook unless you engage with this feature. (Privacy Policy) |
Maven | This supports the Maven widget and search functionality. (Privacy Policy) |
Marketing | |
---|---|
Google AdSense | This is an ad network. (Privacy Policy) |
Google DoubleClick | Google provides ad serving technology and runs an ad network. (Privacy Policy) |
Index Exchange | This is an ad network. (Privacy Policy) |
Sovrn | This is an ad network. (Privacy Policy) |
Facebook Ads | This is an ad network. (Privacy Policy) |
Amazon Unified Ad Marketplace | This is an ad network. (Privacy Policy) |
AppNexus | This is an ad network. (Privacy Policy) |
Openx | This is an ad network. (Privacy Policy) |
Rubicon Project | This is an ad network. (Privacy Policy) |
TripleLift | This is an ad network. (Privacy Policy) |
Say Media | We partner with Say Media to deliver ad campaigns on our sites. (Privacy Policy) |
Remarketing Pixels | We may use remarketing pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to advertise the HubPages Service to people that have visited our sites. |
Conversion Tracking Pixels | We may use conversion tracking pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to identify when an advertisement has successfully resulted in the desired action, such as signing up for the HubPages Service or publishing an article on the HubPages Service. |
Statistics | |
---|---|
Author Google Analytics | This is used to provide traffic data and reports to the authors of articles on the HubPages Service. (Privacy Policy) |
Comscore | ComScore is a media measurement and analytics company providing marketing data and analytics to enterprises, media and advertising agencies, and publishers. Non-consent will result in ComScore only processing obfuscated personal data. (Privacy Policy) |
Amazon Tracking Pixel | Some articles display amazon products as part of the Amazon Affiliate program, this pixel provides traffic statistics for those products (Privacy Policy) |
Clicksco | This is a data management platform studying reader behavior (Privacy Policy) |