Web Search - Googlebot

What is Googlebot ?

googlebot is the crawler bot of Google that search and feed the index of the Google search engine

Articles Related

Rendering

When Googlebot renders a page, it flattens:

• the shadow DOM
• and light DOM content.

Googlebot see only content that's visible in the rendered HTML.

Check it by looking at the rendered HTML with:

• the Mobile-Friendly Test
• or the URL inspection of the Search Console

Ref

How to know if the bot is Google Bot

To verify Googlebot as the caller:

• Run a reverse DNS lookup on the accessing IP address from your logs, using:
  • the host command on linux
  • the nslookup command on windows

nslookup 66.249.66.1

Server:  amplifi.lan
Address:  192.168.135.1

Name:    crawl-66-249-66-1.googlebot.com
Address:  66.249.66.1

• Verify that the domain name is in either googlebot.com or google.com
  • The domain name is crawl-66-249-66-1.googlebot.com : check
• Run a forward DNS lookup on the retrieved domain name using:
  • the host command on linux
  • the nslookup command on windows

nslookup crawl-66-249-66-1.googlebot.com

Server:  amplifi.lan
Address:  192.168.135.1

Non-authoritative answer:
Name:    crawl-66-249-66-1.googlebot.com
Address:  66.249.66.1

• Verify that it is the same as the original accessing IP address from your logs.
  • 66.249.66.1: check !

Documentation / Reference

• GoogleBot reference page
• Verifying Googlebot