12/16/2013 11:35:06 PM

A sample robots.txt file.

*Be careful if using an IDE such as Visual Studio. A byte order marker, or BOM, may be added to the file rendering it unusable by some search engines including Google. If you create the file directly on the server or with Notepad you will avoid this.

User-agent: *
Allow: /
Disallow: /controlpanel/
Disallow: /manage/

User-agent: googlebot        # all services
Disallow: /private/          # disallow this directory
User-agent: googlebot-news   # only the news service
Disallow: /                  # on everything

User-agent: *
Allow: /directory1/myfile.html
Disallow: /directory1/

#disallow everything
User-agent: *
Disallow: /