How damaging are crawl errors reported in Webmaster Tools?

Jump to Last Post 1-6 of 6 discussions (13 posts)
  1. CMHypno profile image93
    CMHypnoposted 8 years ago

    I'm having a wade though Google Webmaster Tools and under crawl errors there were about 20 urls not found and about 60 restricted by robot.txt.

    Is this very damaging and would I be able to make any necessary changes?

    1. IzzyM profile image89
      IzzyMposted 8 years agoin reply to this

      I asked something similar not so long ago, and the general consensus was that it is not damaging because accounts that are surging are also seeing errors. I don't think there is anything we can do about them anyway, as we have no access to the the site html.

      1. CMHypno profile image93
        CMHypnoposted 8 years agoin reply to this

        Thanks Izzy that was my grasp of the day LOL!

  2. Aficionada profile image78
    Aficionadaposted 8 years ago

    I've only seen a few tidbits of discussion about crawl errors - but I know that they were mentioned in the (Matt Cutts) video describing some things that should be checked when traffic falls off.

    What if the problem is (perhaps) the ratio of crawl-error-Hubs to non-error ones?  Has that factor been explored?  If a ratio is involved, it could explain why some surgers see crawl errors too. (That is, if the surgers have crawl errors, but on a very low percentage of their Hubs.)

    And also, even if we ourselves can't do anything about the crawl errors, shouldn't we bring the problem to the attention of the HP staff so the engineers can do something about them - if, in fact, these errors do affect traffic?

    1. IzzyM profile image89
      IzzyMposted 8 years agoin reply to this

      That is something I've been thinking too, Aficionada. My internet is too slow just now to check out a video, but I'd be grateful if someone would take a  look at it  and post a rough synopsis of what he says.

      It's certainly something to look at until we hear something definite to the contrary.

      1. Aficionada profile image78
        Aficionadaposted 8 years agoin reply to this

        IzzyM, it's actually one that I synopsized a couple of weeks ago. … ost1827598

        I didn't understand all of the details then - and still don't; but as I continue to learn more and more, I grasp more about what is being said there.  I don't know whether any of that will help, but there it is.

        Let me know if you see titles of or links to other videos you would like to have a synopsis of.  I'm happy to do that when I have the time; I always learn something in the process.  (PS - My time is fairly limited on weekends now.)

        1. IzzyM profile image89
          IzzyMposted 8 years agoin reply to this

          How embarrassing! I replied in that forum, and don't remember looking anything up - that's my age showing!

          Many thanks, you did a brilliant job!

          I haven't checked Bing and Yahoo, but all my previously popular pages rank really highly in I love that search engine! (that was part of the synopsis - check your rankings in other search engines).

          As for the crawl errors, I am still none the wiser, but as it is controlled at Hubpages end, I will just have to trust that they know what they are doing.

          They are losing out on my account too.

          1. Aficionada profile image78
            Aficionadaposted 8 years agoin reply to this

            I'm giggling because of all the times I have seen some of my own previous posts which I had forgotten about - 1000%!Me too!  You were the one that first mentioned it, as I recall  I'm still working on explanding my use of other search engines, but I have pretty much totally abandoned Google, even for an Image Search.Well, let's hang on here a moment.... I'm going to do a spot of research on previous comments from HP staff, but I don't think we can assume anything about what they do or don't know. (Hasn't previous experience proven that they do make some wild guesses at times? Hmmmm?)  I'll see if I can find out anything and report back later.

  3. IzzyM profile image89
    IzzyMposted 8 years ago

    I'm pretty much sure you won't find anything of any use from HP staff. They stated from the start that some would win from this, and others would lose. Maybe we all pretty much assumed it would be the scrapers and the really poor writers that would suffer, but hey some of us are just accidental casualties.

    I've mostly stopped using Google as a search engine, and not just because their results fail to match what I am searching for.

    Yhey are now a bandwidth hog, and I use up too much of my precious monthly allowance using them.

    I hate how they try to second guess what I type in - "you don't want that, you want this", sort of thing.

    Sometimes they won't let me type in anything - they change my words completely, like text messaging does when it's set to autosuggest.

    Google adverts follow me around like a big lonesome puppydog. OK so I spent several hours researching wifi security because I need to know about it, but now I am looking at other things, and I do not need more adverts for wifi thank you very much.

    Have you seen the new updated Gmail?

    It seems OK, but I think Google should be banned from Google Adsense because that top line advert looks like an email.

    I've nearly clicked it a couple of times - same colors too. Confusing.

    Google Adsense, the new one, takes forever to load. Down rank it for that alone.

    The new Analytics seems OK so far, one out of 3 ain't bad.

  4. Aficionada profile image78
    Aficionadaposted 8 years ago

    Oh dear... I have indeed been researching... and reading... and researching... and reading... and trying to understand.  Where's the smiley that shows me holding both sides of my head in my hands, while little birds chirp and circle around the question marks floating above my dizzy mind?

    I didn't actually read many posts by HP staff - nothing much other than "Don't worry...."  But I did run across some very interesting items from Hubbers, and I have come up with a theory.  Not really a new one - just a reaffirmation of something that I believe is already suspected.

    First, there are several different causes for a crawl error and/or a robots.txt file.  The robots.txt file is used to block the Googlebots from reading a page; someone who knows how to do it can add it manually for legitimate reasons (and it would have to be removed manually in that case - I think); but I think it can also pop up automatically in certain situations.

    One thing that may cause a crawl error is that the server could be down, right at the exact time the bots try to crawl a site; but that error should not show up the next time they crawl. (I'm not sure how likely that is to apply to us here, though.)

    Another problem can occur if there is not enough bandwidth (!), because the bots themselves take up quite a bit of bandwidth.  This caught my eye because I know you have mentioned your bandwidth problem (even before your newest post that I missed while I was trying to sort out these bits of info).  However, I believe that a bots/bandwidth problem would only occur if the servers of HP itself did not have enough bandwidth, so that doesn't seem likely to be the problem. I don't believe your bandwidth issues enter the picture at all.

    Another type of robots.txt problem can happen when a reader uses Google Translate to read the page; but, if I understand correctly, the robots.txt file applies to the translation, not to the original.  (Still, it's something to think about - all those Chinese readers who stole your information?)

    One of the posts in the "Subdomain Switch" mega-forum-thread mentioned that Google does not like sites that have little original content, and G may penalize them, even when the site applies the robots.txt exclusion to prevent crawling. (Post was from Pcunix:  So, what I'm wondering is whether the Chinese scraper sites have ripped off a lot of your other Hubs too, making it appear to Google as though your content is not original, even though it clearly is.

    I ran across this post of yours:
    which made me think that to be even more likely an explanation.

    Right after the subdomain switch, you started getting good traffic and you mentioned that in the forums.  Some lurker (I assume) read about your success  and started ripping you off, posting your Hubs on Chinese sites.  Your DMCA requests were successful in some cases, but Google misread some of the returned reports as counter-claims (which would further lead G to believe that your content was not original, even though it is).  That's how I would explain a large part of what has been happening to you.

    All of these bits of information that I found have raised a lot of questions in my mind, and right now I can't do any more research to answer them.  I don't know whether any of the bits I've posted add anything to the original question.  But there they are.

  5. IzzyM profile image89
    IzzyMposted 8 years ago

    Thank you so much for taking the time to try and help me smile

    If you ever need a favor doing, you know I will oblige if I can.

    You have spent a lot of valuable time researching stuff for me, and all I can say is - wow, I am so glad there are still people like you around!

    I am suspecting that one hub might have a lot to do with my misfortunes, and - well, I'll think about unpublishing it.

    I just hate the thought that when I do, the thieves will be able to claim authorship.

    On the other hand, I researched it, I know the topic inside out, I could easily write another, maybe better, hub, using the same subjects.

    Well worth considering, if that is what is pinning my subdomain down.

    1. Aficionada profile image78
      Aficionadaposted 8 years agoin reply to this

      Well, you're very welcome.  I learn so much when I try to find answers.  The problem comes (I hate to admit it) when I try to regurgitate what I have read and then learn that I misunderstood something and perhaps have steered people wrong!

      I'm just so angry with Google and the way they treat people, and I would love to help find some way authors around the world could give Google the nudge that would put G back on the right track - but it may be way too late for that.

      I have been mulling over your idea of unpublishing your Hub (or rewriting as another Hub).  I keep thinking I have found a way around the problem of losing authorship, but every time that little lightbulb flashes on over my head, it shatters almost immediately.  A good rewrite may be the only solution - published in a different account.

      PS - I don't know that it would matter to Google ever, but if you have any sort of documentation of when you wrote that Hub, be sure to keep a copy - of every bit of documentation or evidence.  (Example - a date stamp, if you wrote it in a word processing file first; anything on a cached copy that would show the date; a screenshot of your stats page showing the date when you published it on HP.)

  6. CMHypno profile image93
    CMHypnoposted 8 years ago

    Izzy and Aficionada, thanks for all this thought and great research. smile  Excuse my absence from my own thread, but I had to go out last night.

    The more I research, the more my head spins, but somehow I will crack it. One thing I am looking at is I had a link to a UK Amazon astore in lots of my hubs, and when in WMT I saw that one hub was showing 2500 links to Amazon - could only be the astore right? (my hubs tend to be longish, but not that long)!

    So removing the astores to see if anything changes - another straw clutched - 'grasp of the day'!


This website uses cookies

As a user in the EEA, your approval is needed on a few things. To provide a better website experience, uses cookies (and other similar technologies) and may collect, process, and share personal data. Please choose which areas of our service you consent to our doing so.

For more information on managing or withdrawing consents and how we handle data, visit our Privacy Policy at:

Show Details
HubPages Device IDThis is used to identify particular browsers or devices when the access the service, and is used for security reasons.
LoginThis is necessary to sign in to the HubPages Service.
Google RecaptchaThis is used to prevent bots and spam. (Privacy Policy)
AkismetThis is used to detect comment spam. (Privacy Policy)
HubPages Google AnalyticsThis is used to provide data on traffic to our website, all personally identifyable data is anonymized. (Privacy Policy)
HubPages Traffic PixelThis is used to collect data on traffic to articles and other pages on our site. Unless you are signed in to a HubPages account, all personally identifiable information is anonymized.
Amazon Web ServicesThis is a cloud services platform that we used to host our service. (Privacy Policy)
CloudflareThis is a cloud CDN service that we use to efficiently deliver files required for our service to operate such as javascript, cascading style sheets, images, and videos. (Privacy Policy)
Google Hosted LibrariesJavascript software libraries such as jQuery are loaded at endpoints on the or domains, for performance and efficiency reasons. (Privacy Policy)
Google Custom SearchThis is feature allows you to search the site. (Privacy Policy)
Google MapsSome articles have Google Maps embedded in them. (Privacy Policy)
Google ChartsThis is used to display charts and graphs on articles and the author center. (Privacy Policy)
Google AdSense Host APIThis service allows you to sign up for or associate a Google AdSense account with HubPages, so that you can earn money from ads on your articles. No data is shared unless you engage with this feature. (Privacy Policy)
Google YouTubeSome articles have YouTube videos embedded in them. (Privacy Policy)
VimeoSome articles have Vimeo videos embedded in them. (Privacy Policy)
PaypalThis is used for a registered author who enrolls in the HubPages Earnings program and requests to be paid via PayPal. No data is shared with Paypal unless you engage with this feature. (Privacy Policy)
Facebook LoginYou can use this to streamline signing up for, or signing in to your Hubpages account. No data is shared with Facebook unless you engage with this feature. (Privacy Policy)
MavenThis supports the Maven widget and search functionality. (Privacy Policy)
Google AdSenseThis is an ad network. (Privacy Policy)
Google DoubleClickGoogle provides ad serving technology and runs an ad network. (Privacy Policy)
Index ExchangeThis is an ad network. (Privacy Policy)
SovrnThis is an ad network. (Privacy Policy)
Facebook AdsThis is an ad network. (Privacy Policy)
Amazon Unified Ad MarketplaceThis is an ad network. (Privacy Policy)
AppNexusThis is an ad network. (Privacy Policy)
OpenxThis is an ad network. (Privacy Policy)
Rubicon ProjectThis is an ad network. (Privacy Policy)
TripleLiftThis is an ad network. (Privacy Policy)
Say MediaWe partner with Say Media to deliver ad campaigns on our sites. (Privacy Policy)
Remarketing PixelsWe may use remarketing pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to advertise the HubPages Service to people that have visited our sites.
Conversion Tracking PixelsWe may use conversion tracking pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to identify when an advertisement has successfully resulted in the desired action, such as signing up for the HubPages Service or publishing an article on the HubPages Service.
Author Google AnalyticsThis is used to provide traffic data and reports to the authors of articles on the HubPages Service. (Privacy Policy)
ComscoreComScore is a media measurement and analytics company providing marketing data and analytics to enterprises, media and advertising agencies, and publishers. Non-consent will result in ComScore only processing obfuscated personal data. (Privacy Policy)
Amazon Tracking PixelSome articles display amazon products as part of the Amazon Affiliate program, this pixel provides traffic statistics for those products (Privacy Policy)
ClickscoThis is a data management platform studying reader behavior (Privacy Policy)