Web - Rate Limiting

A page about rate limiting of HTTP request that is implemented to control bot.

• The HTTP 429 Too Many Requests response status code indicates the user has sent too many requests in a given amount of time (“rate limiting”).
• Web - Robots.txt - Crawl-delay parameters set the number of seconds that the crawler should wait before another crawl.
• The og:ttl object property will limit crawler access (ex: Facebook)