How damaging are crawl errors reported in Webmaster Tools?

1. 84

   CMHypnoposted 12 years ago

   I'm having a wade though Google Webmaster Tools and under crawl errors there were about 20 urls not found and about 60 restricted by robot.txt.
   
   Is this very damaging and would I be able to make any necessary changes?

   replyreport
   1. 88

      IzzyMposted 12 years agoin reply to this

      I asked something similar not so long ago, and the general consensus was that it is not damaging because accounts that are surging are also seeing errors. I don't think there is anything we can do about them anyway, as we have no access to the the site html.

      replyreport
      1. 84

         CMHypnoposted 12 years agoin reply to this

         Thanks Izzy that was my grasp of the day LOL!

         replyreport
2. 80

   Aficionadaposted 12 years ago

   I've only seen a few tidbits of discussion about crawl errors - but I know that they were mentioned in the (Matt Cutts) video describing some things that should be checked when traffic falls off.
   
   What if the problem is (perhaps) the ratio of crawl-error-Hubs to non-error ones?  Has that factor been explored?  If a ratio is involved, it could explain why some surgers see crawl errors too. (That is, if the surgers have crawl errors, but on a very low percentage of their Hubs.)
   
   And also, even if we ourselves can't do anything about the crawl errors, shouldn't we bring the problem to the attention of the HP staff so the engineers can do something about them - if, in fact, these errors do affect traffic?

   replyreport
   1. 88

      IzzyMposted 12 years agoin reply to this

      That is something I've been thinking too, Aficionada. My internet is too slow just now to check out a video, but I'd be grateful if someone would take a  look at it  and post a rough synopsis of what he says.
      
      It's certainly something to look at until we hear something definite to the contrary.

      replyreport
      1. 80

         Aficionadaposted 12 years agoin reply to this

         IzzyM, it's actually one that I synopsized a couple of weeks ago.
         
         http://hubpages.com/forum/topic/84035?p … ost1827598
         
         I didn't understand all of the details then - and still don't; but as I continue to learn more and more, I grasp more about what is being said there.  I don't know whether any of that will help, but there it is.
         
         Let me know if you see titles of or links to other videos you would like to have a synopsis of.  I'm happy to do that when I have the time; I always learn something in the process.  (PS - My time is fairly limited on weekends now.)

         replyreport
         1. 88

            IzzyMposted 12 years agoin reply to this

            How embarrassing! I replied in that forum, and don't remember looking anything up - that's my age showing!
            
            Many thanks, you did a brilliant job!
            
            I haven't checked Bing and Yahoo, but all my previously popular pages rank really highly in Duckduckgo.com. I love that search engine! (that was part of the synopsis - check your rankings in other search engines).
            
            As for the crawl errors, I am still none the wiser, but as it is controlled at Hubpages end, I will just have to trust that they know what they are doing.
            
            They are losing out on my account too.

            replyreport
            1. 80

               Aficionadaposted 12 years agoin reply to this

               I'm giggling because of all the times I have seen some of my own previous posts which I had forgotten about - 1000%!Me too!  You were the one that first mentioned it, as I recall  I'm still working on explanding my use of other search engines, but I have pretty much totally abandoned Google, even for an Image Search.Well, let's hang on here a moment.... I'm going to do a spot of research on previous comments from HP staff, but I don't think we can assume anything about what they do or don't know. (Hasn't previous experience proven that they do make some wild guesses at times? Hmmmm?)  I'll see if I can find out anything and report back later.

               replyreport
3. 88

   IzzyMposted 12 years ago

   I'm pretty much sure you won't find anything of any use from HP staff. They stated from the start that some would win from this, and others would lose. Maybe we all pretty much assumed it would be the scrapers and the really poor writers that would suffer, but hey some of us are just accidental casualties.
   
   I've mostly stopped using Google as a search engine, and not just because their results fail to match what I am searching for.
   
   Yhey are now a bandwidth hog, and I use up too much of my precious monthly allowance using them.
   
   I hate how they try to second guess what I type in - "you don't want that, you want this", sort of thing.
   
   Sometimes they won't let me type in anything - they change my words completely, like text messaging does when it's set to autosuggest.
   
   Google adverts follow me around like a big lonesome puppydog. OK so I spent several hours researching wifi security because I need to know about it, but now I am looking at other things, and I do not need more adverts for wifi thank you very much.
   
   Have you seen the new updated Gmail?
   
   It seems OK, but I think Google should be banned from Google Adsense because that top line advert looks like an email.
   
   I've nearly clicked it a couple of times - same colors too. Confusing.
   
   Google Adsense, the new one, takes forever to load. Down rank it for that alone.
   
   The new Analytics seems OK so far, one out of 3 ain't bad.

   replyreport
4. 80

   Aficionadaposted 12 years ago

   Oh dear... I have indeed been researching... and reading... and researching... and reading... and trying to understand.  Where's the smiley that shows me holding both sides of my head in my hands, while little birds chirp and circle around the question marks floating above my dizzy mind?
   
   I didn't actually read many posts by HP staff - nothing much other than "Don't worry...."  But I did run across some very interesting items from Hubbers, and I have come up with a theory.  Not really a new one - just a reaffirmation of something that I believe is already suspected.
   
   First, there are several different causes for a crawl error and/or a robots.txt file.  The robots.txt file is used to block the Googlebots from reading a page; someone who knows how to do it can add it manually for legitimate reasons (and it would have to be removed manually in that case - I think); but I think it can also pop up automatically in certain situations.
   
   One thing that may cause a crawl error is that the server could be down, right at the exact time the bots try to crawl a site; but that error should not show up the next time they crawl. (I'm not sure how likely that is to apply to us here, though.)
   
   Another problem can occur if there is not enough bandwidth (!), because the bots themselves take up quite a bit of bandwidth.  This caught my eye because I know you have mentioned your bandwidth problem (even before your newest post that I missed while I was trying to sort out these bits of info).  However, I believe that a bots/bandwidth problem would only occur if the servers of HP itself did not have enough bandwidth, so that doesn't seem likely to be the problem. I don't believe your bandwidth issues enter the picture at all.
   
   Another type of robots.txt problem can happen when a reader uses Google Translate to read the page; but, if I understand correctly, the robots.txt file applies to the translation, not to the original.  (Still, it's something to think about - all those Chinese readers who stole your information?)
   
   One of the posts in the "Subdomain Switch" mega-forum-thread mentioned that Google does not like sites that have little original content, and G may penalize them, even when the site applies the robots.txt exclusion to prevent crawling. (Post was from Pcunix: http://hubpages.com/forum/topic/79215#post726248).  So, what I'm wondering is whether the Chinese scraper sites have ripped off a lot of your other Hubs too, making it appear to Google as though your content is not original, even though it clearly is.
   
   I ran across this post of yours:
   http://hubpages.com/forum/topic/86103#post1846319
   which made me think that to be even more likely an explanation.
   
   Right after the subdomain switch, you started getting good traffic and you mentioned that in the forums.  Some lurker (I assume) read about your success  and started ripping you off, posting your Hubs on Chinese sites.  Your DMCA requests were successful in some cases, but Google misread some of the returned reports as counter-claims (which would further lead G to believe that your content was not original, even though it is).  That's how I would explain a large part of what has been happening to you.
   
   All of these bits of information that I found have raised a lot of questions in my mind, and right now I can't do any more research to answer them.  I don't know whether any of the bits I've posted add anything to the original question.  But there they are.

   replyreport
5. 88

   IzzyMposted 12 years ago

   Thank you so much for taking the time to try and help me
   
   If you ever need a favor doing, you know I will oblige if I can.
   
   You have spent a lot of valuable time researching stuff for me, and all I can say is - wow, I am so glad there are still people like you around!
   
   I am suspecting that one hub might have a lot to do with my misfortunes, and - well, I'll think about unpublishing it.
   
   I just hate the thought that when I do, the thieves will be able to claim authorship.
   
   On the other hand, I researched it, I know the topic inside out, I could easily write another, maybe better, hub, using the same subjects.
   
   Well worth considering, if that is what is pinning my subdomain down.

   replyreport
   1. 80

      Aficionadaposted 12 years agoin reply to this

      Well, you're very welcome.  I learn so much when I try to find answers.  The problem comes (I hate to admit it) when I try to regurgitate what I have read and then learn that I misunderstood something and perhaps have steered people wrong!
      
      I'm just so angry with Google and the way they treat people, and I would love to help find some way authors around the world could give Google the nudge that would put G back on the right track - but it may be way too late for that.
      
      I have been mulling over your idea of unpublishing your Hub (or rewriting as another Hub).  I keep thinking I have found a way around the problem of losing authorship, but every time that little lightbulb flashes on over my head, it shatters almost immediately.  A good rewrite may be the only solution - published in a different account.
      
      PS - I don't know that it would matter to Google ever, but if you have any sort of documentation of when you wrote that Hub, be sure to keep a copy - of every bit of documentation or evidence.  (Example - a date stamp, if you wrote it in a word processing file first; anything on a cached copy that would show the date; a screenshot of your stats page showing the date when you published it on HP.)

      replyreport
6. 84

   CMHypnoposted 12 years ago

   Izzy and Aficionada, thanks for all this thought and great research.   Excuse my absence from my own thread, but I had to go out last night.
   
   The more I research, the more my head spins, but somehow I will crack it. One thing I am looking at is I had a link to a UK Amazon astore in lots of my hubs, and when in WMT I saw that one hub was showing 2500 links to Amazon - could only be the astore right? (my hubs tend to be longish, but not that long)!
   
   So removing the astores to see if anything changes - another straw clutched - 'grasp of the day'!

   replyreport