User-agent: * # This is ALL robots
Disallow: /cgi-bin/ # This means NO robot can index my "cgi-bin"
Disallow: /myDirectory/secretPage.html # Or my "secretPage.html" inside "myDirectory".
User-agent: Scooter # This is AltaVista's robot
Disallow: /somePage.asp # This means AltaVista is not allowed to index "somePage.asp", but the rest can.
User-agent: Googlebot-Images # This is Google's image search robot.
Disallow: /myImages/ # This means Google images is not allowed to index my directory "myImages".
Disallow: /myPage-Full-Of-Images.html # Or "myPage-Full-Of-Images.html"
User-agent: WebCrawler # This is WebCrawler's robot
Disallow: / # This means WebCrawler is not allowed to index ANY of my site.
The above example is heavily commented to help you.
Your generated code will not be, however, if you want to add comments,
make sure you use a # sign in front of each comment. The robots will ignore
these comments. Make sure there is 1 empty space in front of and after the # sign.
Personally, I would leave out any comments.
You don't want to make a mistake that could anger any "spiders" and keep them
from indexing your site altogether.
You code will look more like this:
User-agent: *
Disallow: /cgi-bin/
Disallow: /myDirectory/secretPage.html
User-agent: Scooter
Disallow: /somePage.asp
User-agent: Googlebot-Images
Disallow: /myImages/
Disallow: /myPage-Full-Of-Images.html
User-agent: WebCrawler
Disallow: /