Web - Robots.txt

robots.txt is one way to control and gives permission to Web Bot when they crawl your website.

User-agent: *    # applies to all robots
Disallow: /      # disallow indexing of all pages

# Group 1
User-agent: Googlebot
Disallow: /nogooglebot/

# Group 2
User-agent: *
Allow: /

Sitemap: http://www.example.com/sitemap.xml

• Googlebot should not crawl http://www.example.com/nogooglebot and all sub directory
• All other user agents can access the entire site.
• sitemap.xml

The crawl delay is the number of seconds the bot should wait between pageview

Example:

User-agent: *
Disallow: 
Allow: /*

Crawl-delay: 5

See:

• HTML 4.01 Specification - The robots.txt file
• https://developers.google.com/search/reference/robots_txt

An empty value for “Disallow”, indicates that all URIs can be retrieved. At least one “Disallow” field must be present in the robots.txt file.

The “Disallow” field specifies a partial URI that is not to be visited. This can be a full path, or a partial path; any URI that starts with this value will not be retrieved. For example,

• to disallows both /help.html and /help/index.html

Disallow: /help 

• to disallow /help/index.html but allow /help.html.

Disallow: /help/ 

https://www.google.com/webmasters/tools/robots-testing-tool

• http://www.robotstxt.org/
• HTML 4.01 Specification - The robots.txt file
• https://www.imperva.com/blog/most-active-good-bots/