When I do URL submissions, sometimes I get a message saying my robots.txt is blocking the robot crawlers. Also, it says my robots.txt file does not comply with the current “Robots Exclusion Standard” and this may cause problems for some search engines.
Is there anything I can do about this? Or, should I approach HubPages technical support for help? Any advice, please?
This might explain why it is taking forever for me and some others to get our hubs indexed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Looking at Hubpages' robots.txt file (both www.hubpages.com/robots.txt and pinkytoky.hubpages.com/robots.txt) I'm seeing that it blocks the Covario search spiders. Maybe they slow down Hubpages too much?
I don't know, but it's possible some of the places where you're submitting use that particular bot.
That's just a guess. Sounds like a question for tech support.
Thank you very much for the url template. I didn't know we could do that.
Tried contacting HP tech support after reading Greekgeek's reply but got problem getting to their email form. Will try again later. Thanks.
by Quilligrapher 8 years ago
Google AdSense Site Diagnostics reports that a Robots.txt file is blocking their crawler from accessing one of my hubs. Can anyone suggest how I can resolve this issue?
by cashmere 8 years ago
In my site diagnostics in Google adsense these hubs of mine are being shown as blocked. Reason blocked is Robots.txt File.Can you tell me how I can unblock my following hubs?My favourite Tarot layouts,How to teach English and Multilevel marketing - the good the bad and ugly.This has happened since...
by Brinafr3sh 5 years ago
Have any of your Hubs been blocked due to robots.txt file?Did you ever go to one of your Hubs url's and it said "robots.txt file," instead of it showing your hubsite?
by Vibhavari 8 years ago
Hi I was checking my adsense account and in the reports section a number of my URLs appeared as blocked URL and the reason stated was the Robots.txt file. What is a Robots.txt file?Does a blocked URL mean that the clicks on ads on that page are not counted by google adsense? and in the help section...
by Tammy 7 years ago
I tried to asked this question before, but it didn't show up. I logged into Google Webmaster Tools and it said restricted by robots.txt. What is that and how do I fix it? It looks something like this:Show URLs: Restricted by robots.txt (5) URL Detail Detected...
by Mike Bouska 8 years ago
When I logged into Adsense today, it gave me an Analytics diagnostic warning for hubpages, saying that they could not crawl because of a Robots.txt error.Anyone else getting this?
Copyright © 2018 HubPages Inc. and respective owners. Other product and company names shown may be trademarks of their respective owners. HubPages® is a registered Service Mark of HubPages, Inc. HubPages and Hubbers (authors) may earn revenue on this page based on affiliate relationships and advertisements with partners including Amazon, Google, and others.
|HubPages Device ID||This is used to identify particular browsers or devices when the access the service, and is used for security reasons.|
|Login||This is necessary to sign in to the HubPages Service.|
|HubPages Traffic Pixel||This is used to collect data on traffic to articles and other pages on our site. Unless you are signed in to a HubPages account, all personally identifiable information is anonymized.|
|Remarketing Pixels||We may use remarketing pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to advertise the HubPages Service to people that have visited our sites.|
|Conversion Tracking Pixels||We may use conversion tracking pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to identify when an advertisement has successfully resulted in the desired action, such as signing up for the HubPages Service or publishing an article on the HubPages Service.|