HubPages Site Map

1. 63

   Gail Meyersposted 11 years ago

   This was my site map uploaded into my Webmaster Tools.  At roughly the same time I noticed NONE of my hubs were any longer indexed.  The site map is generated by HubPages and was found previously found at:  sitemaps/hub/current/sitemap-hub-gailmeyers.xml.gz#   However, it has since mysteriously disappeared.  So this is directly out of my Webmaster Tools.  I asked a question more than three days ago regarding why this is disallowing.  It appears to me my hub are not being indexed due to the site map created by Hubpages, but my question has been repeatedly ignored for the past three days.
   
   Allow google advertising bot everywhere User-agent: Mediapartners-Google Disallow: # Prevent crawling of pages bots don't need to see User-agent: * Disallow: /email/ Disallow: /c/ Disallow: /docs/ Disallow: /my/ Disallow: /make/request/ Disallow: /profile/*map$ Disallow: /search/ Disallow: /hop/ Disallow: /tip/ Disallow: /answers/ Disallow: /slide/ Disallow: /xml/ User-agent: Covario-IDS Disallow: /
   
   This is a cut and paste directly from Google regarding Google crawlers:
   
   To take another example, say that you want ads on all your pages, but you don't want those pages to appear in Google Search. Here, you'd block Googlebot, but allow Mediapartners-Google, like this: User-agent: Googlebot
   Disallow: /
   
   User-agent: Mediapartners-Google
   Disallow:
   
   I would like an answer from HubPages, because if this is as it appears to be I certainly do not want to invest anymore of my time and energy into creating hubs if HubPages (not Google) is going to prevent them from being indexed when they previously ranked well.

   replyreport
   1. 95

      wildernessposted 11 years agoin reply to this

      Odd.  Searching for how to grow organic celery shows:
      How to grow sweet potatoes
      How to make soap flakes
      How to grow a lemon tree   and
      Organic gardening terms
      all on the first page, all indexed and all your hubs.
      
      The celery hub didn't show up, though.

      replyreport
      1. 63

         Gail Meyersposted 11 years agoin reply to this

         That is odd since I checked them all when I originally posted this question three days ago and none of them were indexed.  At which time the site map was also no longer available.  You do mean in a Google search, right?

         replyreport
         1. 95

            wildernessposted 11 years agoin reply to this

            Yes, I google searched for the URL for the celery hub.
            
            It is possible that all I'm seeing is the dregs, though - that they will be removed from the SERP's in time.  I did look at the cached version of the sweet potato one and it was dated the 16th - well before your discovery.

            replyreport
            1. 63

               Gail Meyersposted 11 years agoin reply to this

               I deleted that site map out of my Webmaster Tools earlier today and at least as of two days ago the site map was no longer at the above address where everyone's site map is supposed to be.  It was in the Webmaster Tools that it said 0 indexed, then I manually checked some of them too.  I did not find any of the ones I spot checked.  I have had a lot of trouble recently with a scraper, which is why I originally started looking into all of this.  Thanks for your response.  I will check everything again.

               replyreport
               1. 95

                  wildernessposted 11 years agoin reply to this

                  Hope it all works out for you.  I've been fighting the scrapers, too, particularly a "Dan Gordon".  I see he's back now and has at least a couple more of mine; back to work on Monday.

                  replyreport
                  1. 63

                     Gail Meyersposted 11 years agoin reply to this

                     Yes, I've seen quite a few hubbers mention the infamous Dan Gordon.  I guess after you file a couple of DMCA reports it becomes routine, but undoubtedly remains annoying.  I have never had any of these issues prior to this, so it seems like a huge, time wasting hassle to me. 
                     
                     My main scraper was a blog, but the deeper I dug the more issues I found.  So I kept looking into every possible cause.  Originally, most of the hubs were still ranking, but not the best ones anymore.  They were what scrapers copied.  So I filed two DCMA reports.  Then I discovered quite a few inbound links from one of the scraper's blogs, as well as numerous other totally unrelated, inbound spam links from various spam pages to my hubs.  So I filed a disavow request listing several spam webpages where the spam links originated.  Then, I ran into this site map issue I have been trying to get a response on.  I am really curious now as to what has happened here.   
                     
                     I hope it all works out for you too.

                     replyreport
                     1. 95

                        wildernessposted 11 years agoin reply to this

                        It's a never ending fight with the thieves.  I've given up on ever think it can be otherwise - it's just a part of running the business of free lancing.  Sad, but it is what it is.
                        
                        My recent spate (and what started those forum posts about "Gordon" was the filing of 30 DMCA's.  All taken down, but two have come right back and one was merely hidden - it's still there.  It will never stop.

                        replyreport
                        1. 63

                           Gail Meyersposted 11 years agoin reply to this

                           It is my understanding "Gordon" goes after those in the Apprentice Program.  I wonder if you are able to tell by the details of this person's actions if that is simple greed or the ugly green monster of jealousy. 
                           
                           I like the copyright infringement notice I saw on a profile that says if you copy his material without permisson that is your acceptance of a contract agreeing to pay him $500 for each violation.  LOL 
                           
                           My main scraper tried to hide it, too.  She cut and pasted other hubber's hubs, too.  The ones at the top of the category under topics.  Then she started making vengeful comments on my hubs after I filed the DCMA.  When I responded to her comment one of my best hubs was unpublished for "hate speech" because I told the scraper she was not only a scraper but had also become an Internet stalker.  Also, Tools or Analytics said someone hit the report button quite a bit.

                           replyreport
                           1. 95

                              wildernessposted 11 years agoin reply to this

                              I've seen Gordon's theft of non AP members; I think HP is just one huge source to him.  Nor does he really seem to USE the stuff, except for black hat SEO purposes - no adsense or other income on any of the sites I found mine on.
                              
                              The one he hid he made it so you couldn't visit the site - had to have passwords or something.  The only way I know it's there is because a search produces the site where it is, not because I can actually see it.
                              
                              You made a forum post about your's didn't you?  I seem to remember that one.  I absolutely hate the thieves - for the most part I won't even ask to have it removed, just go to the host in the hopes they will get tired of seeing it and shut down the whole domain.

                              replyreport
                              1. 63

                                 Gail Meyersposted 11 years agoin reply to this

                                 Yes, when I discovered she had cut and pasted numerous hubs from top to bottom, including ads and comments, I posted it to let the other hubbers know.  She was intentionally trying to outrank the original hubs with black hat.  It was not just the scraping, but also something like 18 inbound links to a hub. 
                                 
                                 The main reason I asked first is because of the topic.  When someone realizes their mother has NPD, the repressed anger often starts to pour out.  Many times someone will start a blog or page to process and share what they learned.  In that scenario, I did not want to discourage or be harsh with someone who has already been through hell if they just did not know any better than to cut and paste the whole thing.  There was one of them who came forward and apologized, she just did not know any better and it was an honest mistaken she took responsibility for.  I can respect that.  However, the main scraper is a completely different story are mentioned above.  I can certainly see in your situation why you would not bother with asking someone to take it down.

                                 replyreport
                           2. 63

                              Gail Meyersposted 11 years agoin reply to this

                              By the way, I looked at one of those NPD hubs just now.  They come up in the search and also say 6 days ago, but the preview shows a picture I changed weeks ago.

                              replyreport
                              1. 95

                                 wildernessposted 11 years agoin reply to this

                                 The ways of Google are inscrutable, aren't they?  That makes no sense at all to me.

                                 replyreport
                                 1. 63

                                    Gail Meyersposted 11 years agoin reply to this

                                    I have been a search engine evaluator for Google, so my opinion of the process is influenced by that.  In my opinion, Google has logical reasons why they do what they do, even if at times no one on the outside looking in can make sense of it.  At the time I evaluated sites, there were about 125 pages to the manual on rating just basic web pages.  There can be human error, but whatever the action it is generally not just an arbitrary action without good reason and being consistently applied.  I am really not frustrated with Google.  Google is a reasonable, although giant beast.  So I know when I clear all of us up, Google will respond accordingly and most likely pretty quickly.  Thank you for your responses in this situation.  I appreciate your assistance.

                                    replyreport
                                    1. 63

                                       Gail Meyersposted 11 years agoin reply to this

                                       "this" not "us"

                                       report
      2. 63

         Gail Meyersposted 11 years agoin reply to this

         That pomegranate one says it was indexed six days ago.

         replyreport
2. STAFF

   Paul Edmondsonposted 11 years ago

   The robots.txt isn't preventing Hubs from getting indexed.  You may want to read about featured hubs.
   
   One other thing that can be a bit frustrating is google says it's normal for not all pages of a site to be indexed.. 
   
   We do our best to create a site hierarchy that emphasizes getting Hubs crawled and indexed.   Creating great pages that others share helps as well.

   replyreport
   1. 68

      Teresa Schultzposted 11 years agoin reply to this

      Oh, thanks for that - I was formulated my long-winded contribution to this forum while you posted yours - your links are helpful, and reading those pages will surely clear up a bit of my own confusion. Thank you.

      replyreport
      1. 68

         Teresa Schultzposted 11 years agoin reply to this

         formulating, not formulated

         replyreport
   2. 63

      Gail Meyersposted 11 years agoin reply to this

      All of my hubs have the "H" beside them in order to be featured.  I'm am not referring to being featured on HubPages.  I am referring to being indexed by Google.  More specifically, the disallow command to the Google bot in the site map:  "User-agent: Mediapartners-Google Disallow:"

      replyreport
      1. 63

         Gail Meyersposted 11 years agoin reply to this

         These hubs were indexed and ranking, then were suddenly nowhere to be found, just as my site map was no long at the web address.  That was as of three, nearly four days ago when I originally posted the question.

         replyreport
         1. 63

            Gail Meyersposted 11 years agoin reply to this

            "nowhere to be found" meaning no longer coming up in Google search results.

            replyreport
      2. 68

         Teresa Schultzposted 11 years agoin reply to this

         Thanks for the clarification - i understood that you meant not being indexed in Google search results, and discussed the featured hubs as felt there was a connection between featured hubs and being indexed on Google - but if all yours do have the "H" I am unable to think what the problem may be - I do hope you find the answers or that HubPages gets back to you, explaining it.

         replyreport
         1. 63

            Gail Meyersposted 11 years agoin reply to this

            Thank you, Teresa.  I appreciate your input.

            replyreport
   3. 63

      Gail Meyersposted 11 years agoin reply to this

      Paul, I have posted the site map HubPages generated, followed by Google's interpretation of the command contained in that site map to disallow User-agent: Mediapartners-Google.  I supported my conclusion or question with the facts of the matter and how I came to that conclusion.  I would appreciate you doing the same.  A statement claiming it is not so because you say it is not so, without any further explanation or support for your statement, really does not suffice at this point.  Then referring me to several resources pages that have nothing to do with my original question, only causes confusion.  It is pretty common knowledge that a site map instructs crawlers and can be instructed not to crawl.  Some of these hubs have been published for more than a year and were previously indexed.  So, it's not quality or a small time gap between between being published and being crawled.

      replyreport
3. 68

   Teresa Schultzposted 11 years ago

   I think that unless a hub is featured (has an "H" next to it in the list of all your hubs in your account, when viewing "my account") it may not be allowed by HubPages to be indexed in search engine results. I'm not sure of this, and I don't think it was always so, but this is what I determine from the words by the "H" next to a title of a hub in one's list of hubs. (See the words by hovering your mouse cursor over that "H." on the right hand side of your list of hubs.)
   
   I say I do not think it was always like this as I think I vaguely recall receiving a HubPages newsletter or something about it some time back.
   
   Also, if I look at the stats of one of my hubs that does not have an "H" next to it in my list of hubs - I just chose a random one published nearly 2 years ago - I see this message: "To boost the odds of having your Hub Featured on related Hubs, Topic Pages, and in search results, consider editing it for mistakes and updating it with fresh, high-quality text and media." Now, I don't know if those "odds" mean that if the hub is of better quality in HubPages' eyes, it would automatically stand a better chance at being indexed in search engine results, or if it means that only then would HubPages actually allow it to be indexed in search engine results.
   
   While looking at the stats of the particular hub I was checking, I see it has had 20 visits from Google dot com, but I don't know if that was before this vaguely remembered newsletter about featured hubs (I don't even remember if hubs always had/didn't have an "H" next to them in my list of hubs) or if those 20 visits from Google dot com were over the entire time my hub has been published.
   
   It is disheartening to think that some of one's hubs may not be allowed (by HubPages) to be indexed by search engines, but at the same time if this is so, HubPages is not really benefitting from these hubs either. I think it would be most disheartening, though, that if after editing a hub and trying to improve it, (as suggested by HubPages in a message when viewing the stats of a hub that does not have an "H") it still doesn't get its "H" - when one doesn't quite know if not having an "H" for a hub means it is not allowed to be indexed by search engines. If we knew for sure, I think we'd all make more of an effort trying to improve existing hubs that don't already have their "H."

   replyreport
4. STAFF

   Paul Edmondsonposted 11 years ago

   User-agent: Mediapartners-Google Disallow: in the robots.txt is the directive that let's AdSense crawl any page with ads.  This helps it contextually match ads to content.  It doesn't prevent google from crawling or indexing your content.

   replyreport
   1. 63

      Gail Meyersposted 11 years agoin reply to this

      Thank you.

      replyreport
5. 53

   Page1 SEO tacticsposted 11 years ago

   If you feel your sitemap is not allowing your articles to be indexed, you should first know that the only things that would prevent your pages from being indexed is a no index tag that un-featured hubs get, or poor stale content (if it's been said once it's been said enough).  However if you are having a problem with getting your pages indexed you can always ask Google webmaster tools to fetch your pages, and once fetched submit them to be indexed, or if you haven't received a sitemap yet due to very few hubs (you need 10+ in order for HP to give you a sitemap) then you can use your feed url to submit as a sitemap and it works just fine.

   replyreport
   1. 63

      Gail Meyersposted 11 years agoin reply to this

      I know a "no index" tag will do it, but I do not know about the above.  That is why I originally asked the question four days ago on another string.  Thank you for your response.  I appreciate it.  I have been fetching the pages only recently since I realized this.  I have plenty of hubs to have a site map.  I previously added the site map to Webmaster Tools.  However, when I noticed the above I went back to find it and it was no longer at the web address.

      replyreport
      1. 53

         Page1 SEO tacticsposted 11 years agoin reply to this

         You're welcome, just remember to to submit your pages to be indexed after fetching otherwise it could still take a while for Google to index them naturally.

         replyreport
         1. 63

            Gail Meyersposted 11 years agoin reply to this

            To follow up, Google responded beautifully to the demotion and disavow requests from all of the spam and black hat links.  Webmaster Tools was also reporting that my top hubs were not linked to my profile, which I fixed by making my name in my copyright notice in the hubs a live link to my profile.  After spending quite a bit of time on clean up, I am happy to report my hubs are now once again reporting as indexed in Webmaster Tools.  That is good news considering Google was previously my main source of traffic, but referral traffic has accounted for 75% of my traffic during this ordeal.

            replyreport
            1. 72

               jacharlessposted 11 years agoin reply to this

               Most of these issues could easily be resolved by integrating the full extend of the cooperative schema. Authors really should not have to manually copyright their work or bounce links around. All that should be already in place so they can focus on writing. And the .gz (sitemap) could be a little more elaborative too. There is so much available to breakout of Breadcrumbs/ the Infinite Horizontal effect...
               Sorry, just typing aloud.
               James.

               replyreport
6. 63

   Gail Meyersposted 11 years ago

   By the way, none of this had anything to do with the quality of the writing in the hubs.

   replyreport