jump to last post 1-3 of 3 discussions (6 posts)

Robots.txt file

  1. pinkytoky profile image86
    pinkytokyposted 5 years ago

    When I do URL submissions, sometimes I get a message saying my robots.txt is blocking the robot crawlers.  Also, it says my robots.txt file does not comply with the current “Robots Exclusion Standard” and this may cause problems for some search engines.

    Is there anything I can do about this?  Or, should I approach HubPages technical support for help?  Any advice, please?

  2. paradigmsearch profile image88
    paradigmsearchposted 5 years ago

    This might explain why it is taking forever for me and some others to get our hubs indexed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  3. Greekgeek profile image95
    Greekgeekposted 5 years ago

    Looking at Hubpages' robots.txt file (both www.hubpages.com/robots.txt and pinkytoky.hubpages.com/robots.txt) I'm seeing that it blocks the Covario search spiders. Maybe they slow down Hubpages too much?

    I don't know, but it's possible some of the places where you're submitting use that particular bot.

    That's just a guess. Sounds like a question for tech support.

    1. paradigmsearch profile image88
      paradigmsearchposted 5 years ago in reply to this

      Thank you very much for the url template. I didn't know we could do that. smile

      1. pinkytoky profile image86
        pinkytokyposted 5 years ago in reply to this

        Tried contacting HP tech support after reading Greekgeek's reply but got problem getting to their email form.  Will try again later.  Thanks.

        1. pinkytoky profile image86
          pinkytokyposted 5 years ago in reply to this

          Just an update.  HP tech team has referred me to http://hubpages.com/faq/#adsenserobots.

 
working