LAN Media

Free SEO tool

Free robots.txt tester

Paste a robots.txt or a homepage. We fetch the file, test your paths, and show the Allow or Disallow line that actually won.

One file fetch. Up to 50 paths against Googlebot, Googlebot-Image, AdsBot, Bingbot, GPTBot, Google-Extended, CCBot and *.

What this tester reports

robots.txt is a crawl filter, not a security control. A bad rule can hide a site from Googlebot, or let a training crawler into a folder you thought was closed.

  • The winning rule. Longest matching path. If Allow and Disallow are the same length, Allow wins.
  • The right group. Most specific user-agent. Googlebot-Image falls back to Googlebot, then *.
  • File issues. Rules before any User-agent, unknown directives, file over 500KB.
  • Directives Google ignores. Crawl-delay, Host, and noindex in robots.txt (since 2019).
  • HTTP meaning. 404 is allow all. 5xx may stop Google crawling the host.
  • Sitemap: lines listed from the file. We do not fetch those XML files here.

How a check works

Paste a homepage, a robots.txt URL, or a draft file. We fetch at most one /robots.txt, then match your paths in memory. Bookmark a check with ?url=.

Limits, on purpose

We do not crawl the site to find blocked URLs, invent Search Console fetch dates, or test 80 bot names. Each run is rate-limited so the next person gets a turn.

Frequently asked questions

What does a robots.txt tester check?

It reads a robots.txt file, then tests paths against Googlebot and a few other agents using Google-style matching: most specific user-agent group, longest path, Allow wins ties.

Did Google remove its robots.txt tester?

Yes, in December 2023. Search Console still has a robots.txt report for fetch errors. This tool tests paths against a file you paste or we fetch.

Does Googlebot-Image use the Googlebot rules?

If there is no Googlebot-Image group, it falls back to Googlebot, then to User-agent: *.

Does Google honour Crawl-delay?

No. Google ignores Crawl-delay, Host, and noindex in robots.txt. Use a meta robots or X-Robots-Tag noindex if you need that.

What does a 404 robots.txt mean?

Allow all. A 5xx response is worse: Google may skip the host until the file is reachable.

This tool is free and logs the requesting IP only to enforce the rate limit. URLs you submit are fetched so we can build the report. See the privacy policy.