Serpkin
Get started

Free tool

Can Google and AI crawlers fetch this URL?

Test a URL against its site's robots.txt for Googlebot, Bingbot and the AI crawlers. See the rule and line that decide, the group that applied, and every problem in the file.

How to get the robots.txt

Enter a page URL, or just the domain to test the homepage. Free, no sign-up; only robots.txt is fetched and nothing is stored.

How robots.txt matching works

Which group applies

One group per crawler
A crawler follows only the group that names it most specifically. Googlebot-Image uses a Googlebot-Image group if there is one, otherwise the Googlebot group.
The * group is the fallback
A crawler that isn't named anywhere follows User-agent: *. A named crawler ignores the * group completely.
Groups are merged
Several groups for the same crawler count as one. User-agent names are case-insensitive.

Which rule wins

The longest match
Of all Allow and Disallow rules that match the path, the longest one decides.
Allow wins a tie
When an Allow and a Disallow rule are equally long, Google picks the less restrictive Allow.
No match means allowed
An empty Disallow: blocks nothing. /robots.txt itself is always allowed.

How paths match

From the start of the path
Disallow: /shop matches /shop, /shop/cart and /shopping. The query string is part of the path.
* and $
* matches any run of characters. $ at the end anchors the end of the URL: /*.pdf$ matches /a.pdf but not /a.pdf?x=1.
Case-sensitive
/Shop and /shop are different paths. Non-ASCII characters are compared percent-encoded.

How Google reads the file

500 KiB limit
Google ignores anything after the first 500 KiB of the file.
Errors change everything
A 4xx (except 429) means no restrictions. A 5xx, a 429 or an unreachable server makes Google stop crawling the site for 12 hours.
Only four fields
User-agent, Allow, Disallow and Sitemap. Google ignores crawl-delay, noindex and other lines.

What the tester flags

  • A robots.txt that returns 5xx, 429, times out or redirects more than five times, and what Google does in each case.
  • An HTML page served as robots.txt, which single-page apps often do.
  • Files over Google's 500 KiB limit, with the line where Google stops reading.
  • Disallow: / for Googlebot or Bingbot, often left over from a staging site.
  • Rules that block the CSS and JavaScript Google needs to render your pages.
  • crawl-delay, noindex and other lines Google ignores, misspelled directives, and rules outside any group.
  • A missing or relative Sitemap line.

The tester follows RFC 9309 and Google's robots.txt documentation. Crawler tokens come from each vendor's own documentation, linked in the results.

Fix what it finds

Questions

How do I test if a URL is blocked by robots.txt?

Enter the full URL above and pick a crawler. The tester fetches that site's robots.txt, finds the group that applies to the crawler, and shows the exact rule and line number that allows or blocks the URL. You can also paste a robots.txt and test it without publishing it. In Search Console, the robots.txt report shows the version Google last fetched.

Does robots.txt stop a page from appearing in Google?

No. robots.txt controls crawling, not indexing. Google can still index a blocked URL, usually without a description, if other pages link to it. To keep a page out of search results, allow crawling and add a noindex robots meta tag or X-Robots-Tag header, or put the page behind a login.

How do I block AI crawlers like GPTBot and ClaudeBot?

Add a group per crawler, for example User-agent: GPTBot with Disallow: /, and the same for ClaudeBot or CCBot. Google-Extended and Applebot-Extended aren't crawlers: blocking them tells Google and Apple not to use your content to train their AI models, without affecting search. Search-type bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether you appear in AI search answers. OpenAI says robots.txt may not apply to ChatGPT-User, because a user starts those visits.

Which rule wins when Allow and Disallow conflict?

The most specific rule, meaning the longest matching path. Allow: /shop/public beats Disallow: /shop for /shop/public/a. If both rules are the same length, Google uses the less restrictive one, Allow. RFC 9309 defines the same order.

What happens if robots.txt returns 404 or 500?

A 404 or any other 4xx except 429 means there are no restrictions: Google crawls everything. A 5xx or 429, a timeout or a DNS error makes Google stop crawling the whole site for 12 hours while it keeps retrying. After that it uses the last good copy for up to 30 days. Google follows at least five redirects for robots.txt before treating it as missing.

Does Google support crawl-delay?

No. Google ignores the crawl-delay line. Some other crawlers read it. If Googlebot puts too much load on your server, returning 503 or 429 for a short time slows it down.