Does HubPages have a robot.txt file limiting crawling of our hubs?
I just started using Google webmaster tools to look at my hubs. I have 26 errors regarding robot.txt. I checked here and it said don't worry about robot.txt. But how can robot.txt not be an issue if Google can't crawl my hubs? I'm sooo confused! Thanks in advance for any light you can shed on this.
If you click on the link "restricted by robots.txt" in google webmaster tools you will see a list of specific URLs that are not being crawled. Restrictions for crawling are for links like getting a widget, asking a question, or for rss feeds. Crawling to your hubs is NOT restricted. Robots.txt can be set up for any website to tell a search crawler not to crawl certain pages.
Thank you, ktrapp! I saw all of those errors on my first use of Google webmaster and tried to find the answer on my own. I found more confusion.
Oh, we newbies! We probably provide the best (and worst) entertainment!
by Mike Bouska8 years ago
When I logged into Adsense today, it gave me an Analytics diagnostic warning for hubpages, saying that they could not crawl because of a Robots.txt error.Anyone else getting this?
by cashmere8 years ago
In my site diagnostics in Google adsense these hubs of mine are being shown as blocked. Reason blocked is Robots.txt File.Can you tell me how I can unblock my following hubs?My favourite Tarot layouts,How to teach...
by Quilligrapher8 years ago
Google AdSense Site Diagnostics reports that a Robots.txt file is blocking their crawler from accessing one of my hubs. Can anyone suggest how I can resolve this issue?
by Ben Guinter5 years ago
I'm curious to hear if there are any hubbers, who saw a big drop in traffic after the most recent Google Panda and Penguin updates, have fixed their issues and are getting good traffic again yet?I think it would be good...
by Peg Cole4 years ago
Is there a way to get rid of unwanted traffic sites that attach a link to our hubs without our knowledge? I try to individually monitor my hubs for suspicious traffic and have written two letters to the offender who has...
by Tammy6 years ago
I tried to asked this question before, but it didn't show up. I logged into Google Webmaster Tools and it said restricted by robots.txt. What is that and how do I fix it? It looks something like this:Show URLs:...
Copyright © 2018 HubPages Inc. and respective owners.
Other product and company names shown may be trademarks of their respective owners.
HubPages® is a registered Service Mark of HubPages, Inc.
HubPages and Hubbers (authors) may earn revenue on this page based on affiliate relationships and advertisements with partners including Amazon, Google, and others.