What Is Googlebot?

Googlebot is the web crawler Google Search uses to find and read your pages before they can be indexed and ranked. It comes in two forms, Googlebot Smartphone and Googlebot Desktop. Google states that most crawl requests now come from the mobile crawler, because it primarily indexes the mobile version of a site. Googlebot fetches a page, then renders it a little later, so what it judges is the finished page, not just the raw code.

What does Googlebot do?

Googlebot discovers and downloads web pages so Google can add them to its index. It follows links from page to page, reads what it finds, and passes the content on to be processed.

Think of it as the reading stage before ranking. Nothing can rank until Googlebot has crawled it and Google has completed indexing it. If Googlebot cannot reach a page, that page is invisible to Search.

Google's own documentation, updated February 2026, names it plainly. Googlebot is "the generic name for two types of web crawlers used by Google Search" (Google Search Central).

It is a Search crawler, and AI access is controlled separately. OpenAI's GPTBot is a distinct crawler. Google-Extended is not a crawler at all but a robots.txt token that governs whether Google's existing crawls may train Gemini, and it does not affect Search, AI Overviews included.

Some AI training data comes from neither. Common Crawl is a free, open archive of the web, run by a nonprofit since 2007. It crawls billions of pages and hands the data to anyone, including the teams that train AI models.

Googlebot Smartphone and Googlebot Desktop

Googlebot works as two crawlers, one simulating a mobile device and one simulating a desktop browser. The mobile crawler now does most of the work.

Google indexes the mobile version of most sites first, so the smartphone crawler makes the majority of crawl requests. The desktop crawler handles a minority. This is why a site that looks fine on a laptop but breaks on a phone can quietly lose ground.

Crawler	What it simulates	Share of crawls
Googlebot Smartphone	a user on a mobile device	the majority
Googlebot Desktop	a user on a desktop browser	a minority

Source: Google Search Central, Googlebot documentation, updated February 2026.

The lesson is simple. Make sure your mobile pages carry the same content and links as your desktop pages, because the mobile version is what Google mostly reads.

Serving different content on purpose is another matter. Cloaking, showing search engines one version of a page and visitors another to win a ranking, is a Google spam violation.

How does Googlebot crawl and render your pages?

Googlebot crawls in stages: it fetches the page first, then renders it separately when Google's resources allow. Rendering is not instant.

Google runs the page's JavaScript with an evergreen version of Chromium, a current build of the same engine behind Chrome (Google Search Central). But the page waits in a render queue first, sometimes for seconds, sometimes longer. This delay is one reason content that appears only after scripts run can be missed, a risk covered in whether JavaScript hurts SEO.

Googlebot also crawls gently. Google says that for most sites it should not access a site more than once every few seconds on average, so you cannot simply demand faster crawling. Google crawls mainly from IP addresses in the United States, and may crawl from other countries if it detects a site blocking US requests.

How do you control and verify Googlebot?

You control Googlebot mainly through your robots.txt file and clear site signals. A blocked file or a missing sitemap can quietly keep pages from being crawled at all.

Allow Googlebot in robots.txt, submit a sitemap, and keep your important pages a short click from the homepage.

You can also confirm a visitor really is Googlebot. Google says to run a reverse DNS lookup and check that the domain resolves to googlebot.com, google.com, or googleusercontent.com.

In our experience, a Malaysian site that slips out of Search has usually gone slow or neglected rather than been hit by anything Googlebot changed. A slow server also tends to be crawled less, so site health is where a crawl check starts.

Frequently asked questions

Is Googlebot the same as Google?

Googlebot is one part of Google, the crawler that fetches pages. It is not the whole search engine. After Googlebot downloads a page, separate Google systems render, index, and rank it. So Googlebot handles discovery and reading, while ranking happens later in the pipeline. When people say "Google crawled my site," the crawler doing it is Googlebot.

Does Googlebot read JavaScript?

Yes, but not immediately. Googlebot renders pages with an evergreen version of Chromium, so it can run JavaScript and see content that scripts add. The catch is timing: rendering happens in a queue after the initial fetch, sometimes considerably later. Content that loads only through JavaScript can be delayed or missed, which is why important text and links are safer in the page's HTML.

How do I know if Googlebot has crawled my site?

Check Google Search Console. The Crawl Stats report shows Googlebot's recent activity, and the URL Inspection tool tells you when a specific page was last crawled. Your server logs also record Googlebot's visits by its user agent. If a page has never been crawled, it will not appear in Search, so crawling is the first thing to confirm when a page is missing.

Can I make Googlebot crawl my site faster?

Not on demand. Google decides its own crawl rate, and says it generally will not hit a site more than once every few seconds. You can help by keeping the site fast, fixing errors, submitting a sitemap, and linking new pages well, all of which make crawling more efficient. You can request a specific page be crawled through URL Inspection, but you cannot force a site-wide speed-up.

How do I block Googlebot from a page?

Use the right tool for the goal. To keep a page out of Google's results, add a noindex rule, not a robots.txt block, because a blocked page can still be listed without its content. To stop crawling entirely, disallow it in robots.txt. Mixing these up is a common mistake, so match the tool to whether you want the page uncrawled or unlisted.

Keep the door open for Googlebot

Storming Solutions runs SEO, AEO, and GEO for Malaysian businesses from Kuala Lumpur, and making sure Googlebot can reach, read, and render every important page is groundwork before any ranking work begins. We would rather fix a crawl block that hides your pages than chase rankings while Google cannot see them.

Worried your pages are not being crawled, or seeing "not indexed" in Search Console? Ask us through the contact page, or read what crawl budget is next. Our SEO, AEO, and GEO service starts by making sure Google can actually see your site.