Web - Robots.txt

robots.txt is a file that control and gives permission to Web Bot when they crawl your website.

User-agent: *    # applies to all robots
Disallow: /      # disallow indexing of all pages

# Group 1
User-agent: Googlebot
Disallow: /nogooglebot/

# Group 2
User-agent: *
Allow: /

Sitemap: http://www.example.com/sitemap.xml

• Googlebot should not crawl http://www.example.com/nogooglebot and all sub directory
• All other user agents can access the entire site.
• sitemap.xml

The crawl delay is the number of seconds the bot should wait between pageview

Example:

User-agent: *
Disallow: 
Allow: /*

Crawl-delay: 5

You can disallow the use of a parameter in a URL query

Example: disallow the do=search for crawler.

User-agent: *
Disallow: /*?*do=search*

The information below comes from:

• the Robots Exclusion Protocol ¹⁾
• the HTML 4.01 Specification ²⁾
• the Google Robot Txt documentation ³⁾

An empty value for “Disallow”, indicates that all URIs can be retrieved. At least one “Disallow” field must be present in the robots.txt file.

The “Disallow” field specifies a partial URI that is not to be visited. This can be a full path, or a partial path; any URI that starts with this value will not be retrieved. For example,

• to disallows both /help.html and /help/index.html

Disallow: /help 

• to disallow /help/index.html but allow /help.html.

Disallow: /help/ 

The pattern allows the following characters ⁴⁾ and is matched against:

• the path
• and the query string

# wildcard
allow: /this/*/exactly
# end of uri
allow: /this/path/exactly$

It the pattern should use one of the above character, it should be % encoded. For example:

The location of the robots.txt is at the root of the endpoint. For instance: https://datacadamia.com/robots.txt

You can test a robots.txt with the robots-testing-tool of google.

^{5) 6)}