Web Search - Googlebot
What is Googlebot ?
googlebot is the crawler bot of Google that search and feed the index of the Google search engine
Articles Related
Rendering
When Googlebot renders a page, it flattens:
- the shadow DOM
- and light DOM content.
Googlebot see only content that's visible in the rendered HTML.
Check it by looking at the rendered HTML with:
- or the URL inspection of the Search Console
Configuration
Crawl Rate
By default, Googlebot will crawl every 90 days (Ref) but you can notify it of a change at the Google Search Console (manually or via the API)
Crawl URL parameters
You can define the parameters in the crawl-url-parameters tools.
How to know if the bot is Google Bot
To verify Googlebot as the caller:
- Run a reverse DNS lookup on the accessing IP address from your logs, using:
- the host command on linux
- the nslookup command on windows
nslookup 66.249.66.1
Server: amplifi.lan
Address: 192.168.135.1
Name: crawl-66-249-66-1.googlebot.com
Address: 66.249.66.1
- Verify that the domain name is in either googlebot.com or google.com
- The domain name is crawl-66-249-66-1.googlebot.com : check
- Run a forward DNS lookup on the retrieved domain name using:
- the host command on linux
- the nslookup command on windows
nslookup crawl-66-249-66-1.googlebot.com
Server: amplifi.lan
Address: 192.168.135.1
Non-authoritative answer:
Name: crawl-66-249-66-1.googlebot.com
Address: 66.249.66.1
- Verify that it is the same as the original accessing IP address from your logs.
- 66.249.66.1: check !