Google AdSense Site Diagnostics reports that a Robots.txt file is blocking their crawler from accessing one of my hubs. Can anyone suggest how I can resolve this issue?
check to see if that particular hub has the checkbox marked that says this hub conforms to google adsense policies. Should be in the field where you set the settings for ads to high low or none and where you change the title or toggle konterra on hubs. Not sure if thats it but its worth a shot to check
Thanks for your prompt reply.
Show Kontera Textlink Ads on this Hub? NOT CHECKED
content meets the conditions of the Google AdSense Program Policy CHECKED
this hub may be considered commerical CHECKED
These selections are the same for all of my hubs.
I do know that those robots . txt thingies show up in the metadata which is searched by search engines so the question will have to be answered by someone more proficient in HTML I suppose. I always thought they had something to do with no follow links but since your author score is way above 75 I don't see how that would block you. I'll leave this one alone and let someone else answer but I would be interested to know the hub url or title or a link, so I can look at the source code and see where that bugger is placed and why ,but i'm not promising an answer.
http://www.robotstxt.org/ this might give you some insight if you don't mind a little technical talk or reading i mean
better yet http://www.google.com/support/webmaster … wer=156449
seems you have a robots,txt file somehow embedded in your meta data of the hub.. the googlebot cannot access your hub until its removed.. basically the .txt file tells all crawlers not to crawl the site. in a text directive which all good crawlers heed the directive in the txt file. Some spammers use crawlewrs that do not heed the .txt file and grab passwrds , etc., but googlebot is a well respected company and their boit heeds the directive in this robots.txt file. The question is how to remove it from the metadata of your hub. You may need to contact staff on this since they have access to change the source code. Was your hub ever flagged by staff recently?
Many thanks for your suggestions, Mel. I am looking into them now.
robots.txt is not embedded anywhere, it is located in the root of the site and controlled by site admins. Send email to the team
Since a couple of months ago I have seen reports about HubPages' robots.txt file blocking Hubs. I was surprised to see it is still going on. I am a webmaster myself and I do my own HTML programming of my site. I can assure you that HubPages does not block individual Hubs. All you have to do is open the robots.txt file in your browser and you'll see it is very innocent. It can always be found in the root directory of any complete website like HubPages.com
As for the meta tags, HubPages places the description meta tag in each hub with the exact text that you enter as the summary of your hub. You are in control of that.
When Google AdSense Site Diagnostics reports that a Robots.txt file is blocking their crawler from accessing one of your hubs, it may be due to a number of reasons not related to HubPages. Either someone clicked on the translate link in Google, or an archived copy of your hub has been residing somewhere. Both of these cases causes a copy of your hub to exist on another server and that server may have a robots.txt file blocking because they don't want the copy to be picked up by search engines. The problem is that your AdSence code is still in the copy, and that triggers the Site Diagnostics report.
Bottom line, I am sure your original Hub is safe and your are getting credit for it. I already experienced a Hub of mine in the Diagnostics. Give it a few days and it clears itself up. This is not a result of anything HUbPages does.
Thank you, Misha. Thank you, Glenn. I appreciate you both taking the time to offer your advice.
The Google code defining the problem follows. It does indeed contain a “translate” URL and flag:
“http:/ / translate. googleusercontent. com/ translate_c? hl= pl&sl= en&u= http:/ / hubpages. com/ hub/ Sendler&prev= / search%3Fq%3Dstefan%2Bzgrzebski%26hl%3Dpl%26lr%3D&rurl= translate. google. pl&usg= ALkJrhg5VqTHa1TEYu9a1BFQbQ41KK0uhg”
Does this support the suggestion that I give this issue a few days to vaporize itself? I was reluctant to contact Hubpages Tech Support because I reasoned that they probably had enough AdSense savvy not to cause an issue of this sort. The last crawl attempt was dated Jan. 02, 2010. Should this blockage problem go away in a week or so?
Again, my sincere thanks for your input.
This issue has been responded to by hub admin in the past. I think ryankett started the thread.
I believe is was explained as a non-issue
The code you posted shows an event where a user used the "google translate" service to read your content. Since adsense is only allowed on pages that are written in english, it would only make sense that it was blocked from being crawled.
other forum posts with this issue
http://hubpages.com/search/include:forums+robots
Quilligrapher, yes it should clear up in a week. I guessed right about it being translated Actually, I had it happen to me and the diagnostic report had cleared the next time I checked in about a week. Please remember that it is ONLY the translated copy that is blocked, not your original hub.
sunforged, thanks for adding the additional info about why translations are blocked.
I am grateful for your input sunforged. I expect the Google AdSense blocked URL comment to vanish in a few days.
Again, I appreciate your taking an interest in my thread, Glenn. I intend to "pay it forward."
i get this too sometimes. the latest one says:
http:/ / www. 123people. com/ ext/ frm? ti= person%20finder&search_term= laura%20buxton&search_country= US&st= person%20finder&target_url= http%3A%2F%2Fsnipsly. com%2F2009%2F12%2F09%2Fplease-return-to-laura-buxton%2F§ion= blog&wrt_id= 262 Robots.txt File Jan 4, 2010
makes absolutely no sense to me, since i am not at 123people or anything, and i don't use snipsly either. they just go away and then other hubs get "blocked", but it's usually only two or three. if it was all of my hubs, then i would wonder.
by cashmere 14 years ago
In my site diagnostics in Google adsense these hubs of mine are being shown as blocked. Reason blocked is Robots.txt File.Can you tell me how I can unblock my following hubs?My favourite Tarot layouts,How to teach English and Multilevel marketing - the good the bad and ugly.This has happened since...
by John Coviello 4 years ago
I have the following message on my Google Adsense account. How can I create an ads.txt file for hubpages.com when I don't control this site? Is this something HubPages will take care of?---------------------------------------------------The message from Adsense:Create an ads.txt file for 1...
by BundleBoy 15 years ago
Instead of fiddling around with Google Adsense and Analytics to figure out which Hub is making me money, Can the team add a new feature to show number of ad clicks for each hub?I am techie and I have been fiddling around for almost 3 days with Google Adsense and Analytics to figure out what is...
by Pamela Lipscomb 11 years ago
I was kind of frustrated with my earning for a while. Since last Feb as a matter of fact. Last month I decided to block Google certified ads and my earnings a up 100 last month. I would be really interested to know how this works for other as well. I also blocked...
by Cindy Lawson 14 years ago
Now don't get me wrong, I totally love writing on Hubpages and being a part of the Hubbing community, but I have lost a little heart recently when one hub of mine on "How to Improve Male Fertility" had the ads disabled based on automatic filters. This problem I reported in order to have...
by geekMachine 9 years ago
Just after being moved from Squidoo, I tried to apply for an AdSense account, which failed for this reason:"Insufficient content: To be approved for AdSense and show relevant ads on your site, your pages need to have enough text on them for our specialists to review and for our crawler to be...
Copyright © 2024 The Arena Media Brands, LLC and respective content providers on this website. HubPages® is a registered trademark of The Arena Platform, Inc. Other product and company names shown may be trademarks of their respective owners. The Arena Media Brands, LLC and respective content providers to this website may receive compensation for some links to products and services on this website.
Copyright © 2024 Maven Media Brands, LLC and respective owners.
As a user in the EEA, your approval is needed on a few things. To provide a better website experience, hubpages.com uses cookies (and other similar technologies) and may collect, process, and share personal data. Please choose which areas of our service you consent to our doing so.
For more information on managing or withdrawing consents and how we handle data, visit our Privacy Policy at: https://corp.maven.io/privacy-policy
Show DetailsNecessary | |
---|---|
HubPages Device ID | This is used to identify particular browsers or devices when the access the service, and is used for security reasons. |
Login | This is necessary to sign in to the HubPages Service. |
Google Recaptcha | This is used to prevent bots and spam. (Privacy Policy) |
Akismet | This is used to detect comment spam. (Privacy Policy) |
HubPages Google Analytics | This is used to provide data on traffic to our website, all personally identifyable data is anonymized. (Privacy Policy) |
HubPages Traffic Pixel | This is used to collect data on traffic to articles and other pages on our site. Unless you are signed in to a HubPages account, all personally identifiable information is anonymized. |
Amazon Web Services | This is a cloud services platform that we used to host our service. (Privacy Policy) |
Cloudflare | This is a cloud CDN service that we use to efficiently deliver files required for our service to operate such as javascript, cascading style sheets, images, and videos. (Privacy Policy) |
Google Hosted Libraries | Javascript software libraries such as jQuery are loaded at endpoints on the googleapis.com or gstatic.com domains, for performance and efficiency reasons. (Privacy Policy) |
Features | |
---|---|
Google Custom Search | This is feature allows you to search the site. (Privacy Policy) |
Google Maps | Some articles have Google Maps embedded in them. (Privacy Policy) |
Google Charts | This is used to display charts and graphs on articles and the author center. (Privacy Policy) |
Google AdSense Host API | This service allows you to sign up for or associate a Google AdSense account with HubPages, so that you can earn money from ads on your articles. No data is shared unless you engage with this feature. (Privacy Policy) |
Google YouTube | Some articles have YouTube videos embedded in them. (Privacy Policy) |
Vimeo | Some articles have Vimeo videos embedded in them. (Privacy Policy) |
Paypal | This is used for a registered author who enrolls in the HubPages Earnings program and requests to be paid via PayPal. No data is shared with Paypal unless you engage with this feature. (Privacy Policy) |
Facebook Login | You can use this to streamline signing up for, or signing in to your Hubpages account. No data is shared with Facebook unless you engage with this feature. (Privacy Policy) |
Maven | This supports the Maven widget and search functionality. (Privacy Policy) |
Marketing | |
---|---|
Google AdSense | This is an ad network. (Privacy Policy) |
Google DoubleClick | Google provides ad serving technology and runs an ad network. (Privacy Policy) |
Index Exchange | This is an ad network. (Privacy Policy) |
Sovrn | This is an ad network. (Privacy Policy) |
Facebook Ads | This is an ad network. (Privacy Policy) |
Amazon Unified Ad Marketplace | This is an ad network. (Privacy Policy) |
AppNexus | This is an ad network. (Privacy Policy) |
Openx | This is an ad network. (Privacy Policy) |
Rubicon Project | This is an ad network. (Privacy Policy) |
TripleLift | This is an ad network. (Privacy Policy) |
Say Media | We partner with Say Media to deliver ad campaigns on our sites. (Privacy Policy) |
Remarketing Pixels | We may use remarketing pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to advertise the HubPages Service to people that have visited our sites. |
Conversion Tracking Pixels | We may use conversion tracking pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to identify when an advertisement has successfully resulted in the desired action, such as signing up for the HubPages Service or publishing an article on the HubPages Service. |
Statistics | |
---|---|
Author Google Analytics | This is used to provide traffic data and reports to the authors of articles on the HubPages Service. (Privacy Policy) |
Comscore | ComScore is a media measurement and analytics company providing marketing data and analytics to enterprises, media and advertising agencies, and publishers. Non-consent will result in ComScore only processing obfuscated personal data. (Privacy Policy) |
Amazon Tracking Pixel | Some articles display amazon products as part of the Amazon Affiliate program, this pixel provides traffic statistics for those products (Privacy Policy) |
Clicksco | This is a data management platform studying reader behavior (Privacy Policy) |