A sample robots.txt file.
*Be careful if using an IDE such as Visual Studio. A byte order marker, or BOM, may be added to the file rendering it unusable by some search engines including Google. If you create the file directly on the server or with Notepad you will avoid this.
#robots.txt User-agent: * Allow: / Disallow: /controlpanel/ Disallow: /manage/ User-agent: googlebot # all services Disallow: /private/ # disallow this directory User-agent: googlebot-news # only the news service Disallow: / # on everything User-agent: * Allow: /directory1/myfile.html Disallow: /directory1/ #disallow everything User-agent: * Disallow: /