A sample robots.txt file.
*Be careful if using an IDE such as Visual Studio. A byte order marker, or BOM, may be added to the file rendering it unusable by some search engines including Google. If you create the file directly on the server or with Notepad you will avoid this.
#robots.txt
User-agent: *
Allow: /
Disallow: /controlpanel/
Disallow: /manage/
User-agent: googlebot # all services
Disallow: /private/ # disallow this directory
User-agent: googlebot-news # only the news service
Disallow: / # on everything
User-agent: *
Allow: /directory1/myfile.html
Disallow: /directory1/
#disallow everything
User-agent: *
Disallow: /