jump to last post 1-3 of 3 discussions (4 posts)

Robot.txt?

  1. profile image82
    Roohi Khanposted 5 years ago

    Can someone tell me what is a robots.txt file is?

    1. Haunty profile image85
      Hauntyposted 5 years ago in reply to this

      It's a text file to tell SE bots which urls not to crawl.
      http://hubpages.com/robots.txt

  2. profile image82
    Roohi Khanposted 5 years ago

    Thanks, Haunty.

    Can you give me an idea of what this means. It is what I found in my webmaster tools robots.txt file for my domain on hubpages.

    If the file is supposed to tell the search engines not to crawl certain pages, wouldn't the information in this file mean that the search engines should not be crawling any of the below given hub sections?
    ________________________

    Sitemap: http://roohikhan.hubpages.com/sitemaps/ … han.xml.gz

    User-agent: Mediapartners-Google
    Disallow:

    User-agent: *
    Disallow: /contact
    Disallow: /hubtivity$
    Disallow: /getwidget
    Disallow: /ask
    Disallow: /fans/
    Disallow: /followers$

    # Google was crawling our custom analytics URLs, so tell them not to
    Disallow: /hub/arts-and-design/
    Disallow: /question/arts-and-design/
    Disallow: /forum/topic/arts-and-design/
    Disallow: /hub/autos/
    Disallow: /question/autos/
    Disallow: /forum/topic/autos/
    Disallow: /hub/books-literature-and-writing/
    Disallow: /question/books-literature-and-writing/
    Disallow: /forum/topic/books-literature-and-writing/
    Disallow: /hub/business-and-employment/
    Disallow: /question/business-and-employment/
    Disallow: /forum/topic/business-and-employment/
    Disallow: /hub/education-and-science/
    Disallow: /question/education-and-science/
    Disallow: /forum/topic/education-and-science/
    Disallow: /hub/entertainment-and-media/
    Disallow: /question/entertainment-and-media/
    Disallow: /forum/topic/entertainment-and-media/
    Disallow: /hub/family-and-parenting/
    Disallow: /question/family-and-parenting/
    Disallow: /forum/topic/family-and-parenting/
    Disallow: /hub/fashion-and-beauty/
    Disallow: /question/fashion-and-beauty/
    Disallow: /forum/topic/fashion-and-beauty/
    Disallow: /hub/food-and-cooking/
    Disallow: /question/food-and-cooking/
    Disallow: /forum/topic/food-and-cooking/
    Disallow: /hub/games-toys-and-hobbies/
    Disallow: /question/games-toys-and-hobbies/
    Disallow: /forum/topic/games-toys-and-hobbies/
    Disallow: /hub/gender-and-relationships/
    Disallow: /question/gender-and-relationships/
    Disallow: /forum/topic/gender-and-relationships/
    Disallow: /hub/health/
    Disallow: /question/health/
    Disallow: /forum/topic/health/
    Disallow: /hub/holidays-and-celebrations/
    Disallow: /question/holidays-and-celebrations/
    Disallow: /forum/topic/holidays-and-celebrations/
    Disallow: /hub/home/
    Disallow: /question/home/
    Disallow: /forum/topic/home/
    Disallow: /hub/hubpages-tutorials-and-community/
    Disallow: /question/hubpages-tutorials-and-community/
    Disallow: /forum/topic/hubpages-tutorials-and-community/
    Disallow: /hub/personal-finance/
    Disallow: /question/personal-finance/
    Disallow: /forum/topic/personal-finance/
    Disallow: /hub/pets-and-animals/
    Disallow: /question/pets-and-animals/
    Disallow: /forum/topic/pets-and-animals/
    Disallow: /hub/politics-and-social-issues/
    Disallow: /question/politics-and-social-issues/
    Disallow: /forum/topic/politics-and-social-issues/
    Disallow: /hub/religion-and-philosophy/
    Disallow: /question/religion-and-philosophy/
    Disallow: /forum/topic/religion-and-philosophy/
    Disallow: /hub/sports-and-recreation/
    Disallow: /question/sports-and-recreation/
    Disallow: /forum/topic/sports-and-recreation/
    Disallow: /hub/technology/
    Disallow: /question/technology/
    Disallow: /forum/topic/technology/
    Disallow: /hub/travel-and-places/
    Disallow: /question/travel-and-places/
    Disallow: /forum/topic/travel-and-places/

    Disallow: /flag/
    Disallow: /email/
    Disallow: /link/
    Disallow: /c/
    Disallow: /docs/
    Disallow: /my/
    Disallow: /make/request/
    Disallow: /profile/*map$
    Disallow: /search/
    Disallow: /widget/
    Disallow: /hop/
    Disallow: /tip/
    Disallow: /fans/
    Disallow: /answers/
    Disallow: /slide/
    Disallow: /signin/
    Disallow: /xml/

    Disallow: /*hot?rss$
    Disallow: /*best?rss$
    Disallow: /*hot/?rss$
    Disallow: /*best/?rss$

    User-agent: Covario-IDS
    Disallow: /

  3. Haunty profile image85
    Hauntyposted 5 years ago

    I don't think so. I wouldn't worry until I find a line that says "Disallow: /hub/*" Why do you ask?

 
working