Hi dusy7969, and all the rest of the people
If you are talking about cPatchs...etc be aware that this is often only for Google Crawler info
And is true that it known to cause duplicates.
The Robots.txt is only a file and has absolut no power over the Spiders or Web Crawlers, they only respect what you have in the "text document" if they want, and since they belong to diferent companies and organizations they woul not acept the others rules.
I do not know where you host your pages and if you can edit and use any html code inside your web pages, but one thing that all or almost all Web Crawlers "obey" is to the Metatags that you insert in the Html Head's document.
If you know how to use it then i will show you where you can generate them easy
If you do not know, you can always use the same link from SINALSOFT where you can generate your MetaTags in less than 1 minute (freeware - FreeOnlineTools)
here: ( http://www.sinalsoft.com/tools/?lg=en&com=&... )
You can also read my 3 Meta Tags Articles Here: ( https://hubpages.com/technology/programming-a-baby... )
I hope this helps