Is the robot.txt disllowing as intended?

  1. Howard S. profile image86
    Howard S.posted 4 years ago

    Robot.txt disallows the following for every topic:
    Disallow: /forum/topic/autos/

    I don't believe that actually stops anything. I believe what is really intended is:
    Disallow: /topic/autos/

    We disallow /search/. That's good. What about /tag/?

  2. Howard S. profile image86
    Howard S.posted 4 years ago

    I believe I made an error there. You may be trying to exclude one of two things, but neither one works. The following would each work.

    In the forums, they are called category rather than topics:
    Disallow: /forum/category/autos/

    My correction outside the forums inadvertently ommitted the s from topics:
    Disallow: /topics/autos/

 
working