Robots.txt Example

12/16/2013 11:35:06 PM

A sample robots.txt file.

*Be careful if using an IDE such as Visual Studio. A byte order marker, or BOM, may be added to the file rendering it unusable by some search engines including Google. If you create the file directly on the server or with Notepad you will avoid this.

#robots.txt
User-agent: *
Allow: /
Disallow: /controlpanel/
Disallow: /manage/

User-agent: googlebot        # all services
Disallow: /private/          # disallow this directory
 
User-agent: googlebot-news   # only the news service
Disallow: /                  # on everything

User-agent: *
Allow: /directory1/myfile.html
Disallow: /directory1/

#disallow everything
User-agent: *
Disallow: /