jump to last post 1-6 of 6 discussions (34 posts)

HubPages Site Map

  1. Gail Meyers profile image59
    Gail Meyersposted 4 years ago

    This was my site map uploaded into my Webmaster Tools.  At roughly the same time I noticed NONE of my hubs were any longer indexed.  The site map is generated by HubPages and was found previously found at:  sitemaps/hub/current/sitemap-hub-gailmeyers.xml.gz#   However, it has since mysteriously disappeared.  So this is directly out of my Webmaster Tools.  I asked a question more than three days ago regarding why this is disallowing.  It appears to me my hub are not being indexed due to the site map created by Hubpages, but my question has been repeatedly ignored for the past three days.

    Allow google advertising bot everywhere User-agent: Mediapartners-Google Disallow: # Prevent crawling of pages bots don't need to see User-agent: * Disallow: /email/ Disallow: /c/ Disallow: /docs/ Disallow: /my/ Disallow: /make/request/ Disallow: /profile/*map$ Disallow: /search/ Disallow: /hop/ Disallow: /tip/ Disallow: /answers/ Disallow: /slide/ Disallow: /xml/ User-agent: Covario-IDS Disallow: /

    This is a cut and paste directly from Google regarding Google crawlers:

    To take another example, say that you want ads on all your pages, but you don't want those pages to appear in Google Search. Here, you'd block Googlebot, but allow Mediapartners-Google, like this: User-agent: Googlebot
    Disallow: /

    User-agent: Mediapartners-Google
    Disallow:

    I would like an answer from HubPages, because if this is as it appears to be I certainly do not want to invest anymore of my time and energy into creating hubs if HubPages (not Google) is going to prevent them from being indexed when they previously ranked well.

    1. wilderness profile image95
      wildernessposted 4 years ago in reply to this

      Odd.  Searching for how to grow organic celery shows:
      How to grow sweet potatoes
      How to make soap flakes
      How to grow a lemon tree   and
      Organic gardening terms
      all on the first page, all indexed and all your hubs.

      The celery hub didn't show up, though.

      1. Gail Meyers profile image59
        Gail Meyersposted 4 years ago in reply to this

        That is odd since I checked them all when I originally posted this question three days ago and none of them were indexed.  At which time the site map was also no longer available.  You do mean in a Google search, right?

        1. wilderness profile image95
          wildernessposted 4 years ago in reply to this

          Yes, I google searched for the URL for the celery hub.

          It is possible that all I'm seeing is the dregs, though - that they will be removed from the SERP's in time.  I did look at the cached version of the sweet potato one and it was dated the 16th - well before your discovery.

          1. Gail Meyers profile image59
            Gail Meyersposted 4 years ago in reply to this

            I deleted that site map out of my Webmaster Tools earlier today and at least as of two days ago the site map was no longer at the above address where everyone's site map is supposed to be.  It was in the Webmaster Tools that it said 0 indexed, then I manually checked some of them too.  I did not find any of the ones I spot checked.  I have had a lot of trouble recently with a scraper, which is why I originally started looking into all of this.  Thanks for your response.  I will check everything again.

            1. wilderness profile image95
              wildernessposted 4 years ago in reply to this

              Hope it all works out for you.  I've been fighting the scrapers, too, particularly a "Dan Gordon".  I see he's back now and has at least a couple more of mine; back to work on Monday.

              1. Gail Meyers profile image59
                Gail Meyersposted 4 years ago in reply to this

                Yes, I've seen quite a few hubbers mention the infamous Dan Gordon.  I guess after you file a couple of DMCA reports it becomes routine, but undoubtedly remains annoying.  I have never had any of these issues prior to this, so it seems like a huge, time wasting hassle to me. 

                My main scraper was a blog, but the deeper I dug the more issues I found.  So I kept looking into every possible cause.  Originally, most of the hubs were still ranking, but not the best ones anymore.  They were what scrapers copied.  So I filed two DCMA reports.  Then I discovered quite a few inbound links from one of the scraper's blogs, as well as numerous other totally unrelated, inbound spam links from various spam pages to my hubs.  So I filed a disavow request listing several spam webpages where the spam links originated.  Then, I ran into this site map issue I have been trying to get a response on.  I am really curious now as to what has happened here.   

                I hope it all works out for you too.

                1. wilderness profile image95
                  wildernessposted 4 years ago in reply to this

                  It's a never ending fight with the thieves.  I've given up on ever think it can be otherwise - it's just a part of running the business of free lancing.  Sad, but it is what it is.

                  My recent spate (and what started those forum posts about "Gordon" was the filing of 30 DMCA's.  All taken down, but two have come right back and one was merely hidden - it's still there.  It will never stop.

                  1. Gail Meyers profile image59
                    Gail Meyersposted 4 years ago in reply to this

                    It is my understanding "Gordon" goes after those in the Apprentice Program.  I wonder if you are able to tell by the details of this person's actions if that is simple greed or the ugly green monster of jealousy. 

                    I like the copyright infringement notice I saw on a profile that says if you copy his material without permisson that is your acceptance of a contract agreeing to pay him $500 for each violation.  LOL 

                    My main scraper tried to hide it, too.  She cut and pasted other hubber's hubs, too.  The ones at the top of the category under topics.  Then she started making vengeful comments on my hubs after I filed the DCMA.  When I responded to her comment one of my best hubs was unpublished for "hate speech" because I told the scraper she was not only a scraper but had also become an Internet stalker.  Also, Tools or Analytics said someone hit the report button quite a bit.

      2. Gail Meyers profile image59
        Gail Meyersposted 4 years ago in reply to this

        That pomegranate one says it was indexed six days ago.

  2. Paul Edmondson profile image
    88
    Paul Edmondsonposted 4 years ago

    The robots.txt isn't preventing Hubs from getting indexed.  You may want to read about featured hubs.

    One other thing that can be a bit frustrating is google says it's normal for not all pages of a site to be indexed.

    We do our best to create a site hierarchy that emphasizes getting Hubs crawled and indexed.   Creating great pages that others share helps as well.

    1. Teresa Schultz profile image84
      Teresa Schultzposted 4 years ago in reply to this

      Oh, thanks for that - I was formulated my long-winded contribution to this forum while you posted yours - your links are helpful, and reading those pages will surely clear up a bit of my own confusion. Thank you.

      1. Teresa Schultz profile image84
        Teresa Schultzposted 4 years ago in reply to this

        formulating, not formulated

    2. Gail Meyers profile image59
      Gail Meyersposted 4 years ago in reply to this

      All of my hubs have the "H" beside them in order to be featured.  I'm am not referring to being featured on HubPages.  I am referring to being indexed by Google.  More specifically, the disallow command to the Google bot in the site map:  "User-agent: Mediapartners-Google Disallow:"

      1. Gail Meyers profile image59
        Gail Meyersposted 4 years ago in reply to this

        These hubs were indexed and ranking, then were suddenly nowhere to be found, just as my site map was no long at the web address.  That was as of three, nearly four days ago when I originally posted the question.

        1. Gail Meyers profile image59
          Gail Meyersposted 4 years ago in reply to this

          "nowhere to be found" meaning no longer coming up in Google search results.

      2. Teresa Schultz profile image84
        Teresa Schultzposted 4 years ago in reply to this

        Thanks for the clarification - i understood that you meant not being indexed in Google search results, and discussed the featured hubs as felt there was a connection between featured hubs and being indexed on Google - but if all yours do have the "H" I am unable to think what the problem may be - I do hope you find the answers or that HubPages gets back to you, explaining it.

        1. Gail Meyers profile image59
          Gail Meyersposted 4 years ago in reply to this

          Thank you, Teresa.  I appreciate your input.

    3. Gail Meyers profile image59
      Gail Meyersposted 4 years ago in reply to this

      Paul, I have posted the site map HubPages generated, followed by Google's interpretation of the command contained in that site map to disallow User-agent: Mediapartners-Google.  I supported my conclusion or question with the facts of the matter and how I came to that conclusion.  I would appreciate you doing the same.  A statement claiming it is not so because you say it is not so, without any further explanation or support for your statement, really does not suffice at this point.  Then referring me to several resources pages that have nothing to do with my original question, only causes confusion.  It is pretty common knowledge that a site map instructs crawlers and can be instructed not to crawl.  Some of these hubs have been published for more than a year and were previously indexed.  So, it's not quality or a small time gap between between being published and being crawled.

  3. Teresa Schultz profile image84
    Teresa Schultzposted 4 years ago

    I think that unless a hub is featured (has an "H" next to it in the list of all your hubs in your account, when viewing "my account") it may not be allowed by HubPages to be indexed in search engine results. I'm not sure of this, and I don't think it was always so, but this is what I determine from the words by the "H" next to a title of a hub in one's list of hubs. (See the words by hovering your mouse cursor over that "H." on the right hand side of your list of hubs.)

    I say I do not think it was always like this as I think I vaguely recall receiving a HubPages newsletter or something about it some time back.

    Also, if I look at the stats of one of my hubs that does not have an "H" next to it in my list of hubs - I just chose a random one published nearly 2 years ago - I see this message: "To boost the odds of having your Hub Featured on related Hubs, Topic Pages, and in search results, consider editing it for mistakes and updating it with fresh, high-quality text and media." Now, I don't know if those "odds" mean that if the hub is of better quality in HubPages' eyes, it would automatically stand a better chance at being indexed in search engine results, or if it means that only then would HubPages actually allow it to be indexed in search engine results.

    While looking at the stats of the particular hub I was checking, I see it has had 20 visits from Google dot com, but I don't know if that was before this vaguely remembered newsletter about featured hubs (I don't even remember if hubs always had/didn't have an "H" next to them in my list of hubs) or if those 20 visits from Google dot com were over the entire time my hub has been published.

    It is disheartening to think that some of one's hubs may not be allowed (by HubPages) to be indexed by search engines, but at the same time if this is so, HubPages is not really benefitting from these hubs either. I think it would be most disheartening, though, that if after editing a hub and trying to improve it, (as suggested by HubPages in a message when viewing the stats of a hub that does not have an "H") it still doesn't get its "H" - when one doesn't quite know if not having an "H" for a hub means it is not allowed to be indexed by search engines. If we knew for sure, I think we'd all make more of an effort trying to improve existing hubs that don't already have their "H."

  4. Paul Edmondson profile image
    88
    Paul Edmondsonposted 4 years ago

    User-agent: Mediapartners-Google Disallow: in the robots.txt is the directive that let's AdSense crawl any page with ads.  This helps it contextually match ads to content.  It doesn't prevent google from crawling or indexing your content.

    1. Gail Meyers profile image59
      Gail Meyersposted 4 years ago in reply to this

      Thank you.

  5. Page1 SEO tactics profile image78
    Page1 SEO tacticsposted 4 years ago

    If you feel your sitemap is not allowing your articles to be indexed, you should first know that the only things that would prevent your pages from being indexed is a no index tag that un-featured hubs get, or poor stale content (if it's been said once it's been said enough).  However if you are having a problem with getting your pages indexed you can always ask Google webmaster tools to fetch your pages, and once fetched submit them to be indexed, or if you haven't received a sitemap yet due to very few hubs (you need 10+ in order for HP to give you a sitemap) then you can use your feed url to submit as a sitemap and it works just fine.

    1. Gail Meyers profile image59
      Gail Meyersposted 4 years ago in reply to this

      I know a "no index" tag will do it, but I do not know about the above.  That is why I originally asked the question four days ago on another string.  Thank you for your response.  I appreciate it.  I have been fetching the pages only recently since I realized this.  I have plenty of hubs to have a site map.  I previously added the site map to Webmaster Tools.  However, when I noticed the above I went back to find it and it was no longer at the web address.

      1. Page1 SEO tactics profile image78
        Page1 SEO tacticsposted 4 years ago in reply to this

        You're welcome, just remember to to submit your pages to be indexed after fetching otherwise it could still take a while for Google to index them naturally.

        1. Gail Meyers profile image59
          Gail Meyersposted 4 years ago in reply to this

          To follow up, Google responded beautifully to the demotion and disavow requests from all of the spam and black hat links.  Webmaster Tools was also reporting that my top hubs were not linked to my profile, which I fixed by making my name in my copyright notice in the hubs a live link to my profile.  After spending quite a bit of time on clean up, I am happy to report my hubs are now once again reporting as indexed in Webmaster Tools.  That is good news considering Google was previously my main source of traffic, but referral traffic has accounted for 75% of my traffic during this ordeal.

          1. jacharless profile image77
            jacharlessposted 4 years ago in reply to this

            Most of these issues could easily be resolved by integrating the full extend of the cooperative schema. Authors really should not have to manually copyright their work or bounce links around. All that should be already in place so they can focus on writing. And the .gz (sitemap) could be a little more elaborative too. There is so much available to breakout of Breadcrumbs/ the Infinite Horizontal effect...
            Sorry, just typing aloud.
            James.

  6. Gail Meyers profile image59
    Gail Meyersposted 4 years ago

    By the way, none of this had anything to do with the quality of the writing in the hubs.

 
working