I have a hub that went unfeatured several days back. It was my second hub, and it really was in dire need of rewriting. After doing an extensive revision, I sent it out to the QAP. Then I checked the stats to see how many views it has gotten to date.
I was surprised to see that a Google bot had crawled it two hours ago, and a Bing bot had crawled it 12 hours ago. Aren't unfeatured hubs supposed to be "unseen" by the search engines?
Any idea why the bot crawls? This isn't a huge issue, but I am curious. The hub is now listed as "pending."
How could you tell that the bots were crawling?
Crawling and being indexed in a search engine are different things.
Okay, I am being crawled; how do I find out if I am indexed?
By searching for your Hub in the Search Engine in which you would like to determine indexed status.
So if I was trying to see if this Hub is indexed in Google:
http://hubpages.com/learningcenter/Diag … r-problems
You would search this exact string in Google:
So the resulting search is:
https://www.google.com/#hl=en&sclie … mp;bih=576
I generally search for the URL. It's almost always on the first line, sometimes the second. Either way, if it's there it is indexed.
They're unseen but not unvisted. Spiders can and will follow noindex links, they just won't put them on search engines.
Bots are always crawling to find new and updated, fresh content.
When you check hub stats at the top of each hub, click Search and it displays when search engines last crawled that page.
Wow, thanks! A really valuable bit of info.
I'm aware that Google bots are always crawling the web. And I went to the stats tab to find the bot had crawled, but it crawled a hub that had been deindexed. I guess this must be normal, but I thought Google and other engines couldn't see our deindexed (unfeatured) hubs. It's not important, however I am curious.
A page has to be crawled in order for a bot to 'see' the noindex tag.
Sometimes a no-indexed page may still show in results if a bot hasn't crawled your site since the tag was added.
That is surprising. It's good to know that it can be happening, but still a bit confusing.
Janie, did you check your hubs at the top of the page under "stats"? I didn't know my Hubs had been crawled, but I guess they are.
Don't see that this has happened to any of my hubs to date. It is certainly a concern.
Have you signed up for Google Plus? If you get a gmail account you can sign up from there. That might help. There's also a "google authorship" hub floating around that I'll find, and I'll send you the url.
Jane, I see you've only been here for 2 months. Sometimes it takes a while for a page to get ranked and to receive frequent crawls.
I looked at your profile page and see that you have some niche topics that you're writing about. I highly recommend linking your hubs together. You can select a keyword, like 'sled dog' in your hubs and insert a link to another 'sled dog' hub. You can link your berry articles, and your pet therapy articles. Linking your sub-domain will help and it also helps with establishing relevancy and authority.
This is a link to a series of articles that I found helpful trying to understand how search engines work. http://www.seomoz.org/beginners-guide-t … es-operate
And yet with the nature of the giant beast that the internet has become, search is always evolving.
Essentially, the bot sees nothing important if it is idled.
OK. So it will crawl, see nothing, go away and not "index" the hub. But it can still crawl the hub.
Now that I've had some hubs have the no-follow tag when first crawled I'm beginning to suspect even when the hubs are finally indexed they're not treated the same as if the no-follow tag had never been seen on the hub to begin with. I really hope HP can soon get the vetting process done quicker for proven writers before the bots see a noindex-tag on a hub.
But.....and there is a but, if it sees the no index tag it may not care to come back for a long while after the hub is edited. Also, some people---and I am included in this camp--suspect this tag has a long term detrimental effect on the particular hub.
Most of the time when a hub is idled it never gets much Google traffic ever again. Thus all of the controversy surrounding the new program. Not to mention when a hub is idled you lose any organic back links which the hub has gained during its featured phase. Too many possible bad things versus an unknown good, in my opinion.
Thank you. That helps me understand why people get upset when a hub goes idle. Perhaps there is a way to give the hub a whole new address, if the hubber so desires, to make Google look at it as "new." Back links will still be lost, however.
Yes, I've been concerned about this too. A hub could be already indexed, get idled and the no-follow tag, then get crawled. And I've been wondering what effect that has on the hub. I've wondered if it affected indexing. This seems especially relevant since it has been said that a hub (or any content) must age to start getting indexed and do well in search results, etc.
Edit: I think it would help to know exactly what effect the no-follow tag has; it means it doesn't index. Then is it that it is seen as something not to index, by crawlers? And does it affect indexing that's already been done? If so, it would seem like erasing work done and age gained.
I thought it just meant they were still crawled but not indexed in searches
I've been waiting 21 hours now for my latest hub to be featured. So much for the BS about times getting shorter. And Simone wonders why they have the rep they do for not being trustworthy. DUH!
I've been waiting about 18 or 19 hours or so for my latest hub to get featured. It got crawled by Google during that time, and it had a no index tag on it.
However, I also posted the hub up on FB from which most of my traffic comes. As this hub is going to be a "timely" hub and not evergreen, I needed it up and running quickly. Google hits will matter little to nothing for this particular hub. It should get hits from being "passed around" the target community.
That being said, if this hub were "timely" to a news story that I needed Google hits off of, it would have failed at this point. I'm starting to see why some people worry about the long process for the QAP. I honestly understand the need for a QA program after reading a lot of poor hubs, but I am hoping HP can come up with something a bit faster for its good writers. (BTW, I'm honestly not sure I qualify for the "good writers" category, but I know others that would. )
Based on the nature of this thread, dealing with Google crawling unindexed Hubs, I thought I'd add some interesting additional information.
As was correctly explained by several people above, Google needs to keep crawling to check if and when the noindex tag is removed so that it can index the hub again.
But I have discovered that Google also keeps crawling deleted URLs as well. How do I know this? Because I have my own business site where I can play with my own code.
I wrote code in my 404 error page that shows me all the traffic of people (or bots) looking for non-existent pages. My report also shows me exactly what they are looking for -- the exact URL.
I usually find hackers this way. They are looking for back doors to log in and do harm. I can then block them.
But the point I want to make here is that I also see Google bots crawling pages that I had deleted -- even years ago. For some reason, I guess, Google finds it important to keep checking if an old deleted page will ever be recreated.
So, the bottom line is that Google will keep crawling anything that ever existed, if it was set to "noindex" or even if it was entirely deleted.
Of course, I am not addressing the big issue -- a delay that may be imposed by noindexing. This all depends on how long it takes for Google to re-crawl, and then how long it takes for them to re-index after finding the that the noindex tag was removed. That timing all seems to be different with each website. And maybe even with each subdomain.
by west405 years ago
I have made some changes in some of my hubs and am wondering if I should resubmit these url's or if they will automatically be "crawled" and I do not need to resubmit. Any help will be appreciated and I...
by Simone Haruko Smith4 years ago
We have added a feature to the Search tab under the stats box on each Hub that enables you to see when that Hub that has last been crawled by search engines.When an Idle Hub is crawled, search engines will see that it...
by David Stillwell4 years ago
For the first time, after publishing a new hub, I was not able to find it in google search for three days. I had to manually enter the hubs information into google so that it would be added to the search engines. ...
by David Stillwell4 years ago
Staff@hubpages.com:So I published my second hub in a very long time. The last hub publised was not crawled by google for days on end. I ended up having to manually submit it. Fine! A few days ago, I published a...
by Gary Anderson17 months ago
I thought if a hubpage was listed in Google search it was featured. But apparently that is not the truth. I guess I will keep the hubs up and run my own check on them.
by Jacob J Morris2 years ago
I'm in the process of migrating my content from my blog to HubPages and just transferred an article titled "Top 50 Men's Grooming Blogs for 2014." I just switched this article to HubPages a few moments ago...
Copyright © 2016 HubPages Inc. and respective owners.
Other product and company names shown may be trademarks of their respective owners.
HubPages® is a registered Service Mark of HubPages, Inc.
HubPages and Hubbers (authors) may earn revenue on this page based on affiliate relationships and advertisements with partners including Amazon, Google, and others.