Table of Contents

What is Googlebot ?

googlebot is the crawler bot of Google that search and feed the index of the Google search engine

Rendering

When Googlebot renders a page, it flattens:

Googlebot see only content that's visible in the rendered HTML.

Check it by looking at the rendered HTML with:

Url Inspection Google Search Screenshot

Ref

Configuration

Crawl Rate

By default, Googlebot will crawl every 90 days (Ref) but you can notify it of a change at the Google Search Console (manually or via the API)

Crawl URL parameters

You can define the parameters in the crawl-url-parameters tools.

How to know if the bot is Google Bot

To verify Googlebot as the caller: 

nslookup 66.249.66.1

Server:  amplifi.lan
Address:  192.168.135.1

Name:    crawl-66-249-66-1.googlebot.com
Address:  66.249.66.1
  • Verify that the domain name is in either googlebot.com or google.com
    • The domain name is crawl-66-249-66-1.googlebot.com : check
  • Run a forward DNS lookup on the retrieved domain name using:
    • the host command on linux
    • the nslookup command on windows
nslookup crawl-66-249-66-1.googlebot.com

Server:  amplifi.lan
Address:  192.168.135.1

Non-authoritative answer:
Name:    crawl-66-249-66-1.googlebot.com
Address:  66.249.66.1
  • Verify that it is the same as the original accessing IP address from your logs.
    • 66.249.66.1: check !

Documentation / Reference