Robots.txt Tester

A robots.txt tester downloads the robots.txt of a site and checks whether a specific URL is allowed or blocked for a user agent such as Googlebot. Robots.txt controls crawling, not indexing.

This tool fetches the address you enter once, from our server, and reports what a crawler would receive. The URL is not stored; only a short-lived rate-limit counter is kept. Results show the page at the moment of the check and can differ from what a logged-in browser or a bot with different headers sees.

How to read the result and act on it

The result tells you whether the crawler you chose may fetch the URL, and which rule decided it. If an important page is blocked, find the matching Disallow line and narrow or remove it, keeping in mind that the most specific matching rule wins and Allow can override a broader Disallow. If a page you want kept out is allowed, remember robots.txt controls crawling, not indexing, so use noindex on a crawlable page instead. Test the main templates of your site: home, category, product, search results and assets such as CSS and JavaScript. Retest after each edit, and confirm the file is served at the root with a 200 status.

Common mistakes to avoid

A single stray slash can block an entire site, as in Disallow: /, so always retest after edits. Blocking CSS and JavaScript folders can stop search engines from rendering pages properly. Another mistake is relying on robots.txt to hide sensitive content; the file is public and lists the paths you want to hide. Do not combine a Disallow with a noindex tag on the same URL, because the crawler never fetches the page to see the tag. Pay attention to case, since paths are case sensitive, and to user agent groups, because a crawler follows only the most specific group that matches it.

Does robots.txt stop a page from being indexed?

Not reliably. A blocked URL can still be indexed from links without its content. Use noindex on a crawlable page to prevent indexing.

What is the longest-match rule?

When Allow and Disallow rules both match, the most specific (longest) rule wins.

Should I block AI crawlers in robots.txt?

That is a business decision. Blocking can limit use of your content by those systems, and it may reduce visibility in their products.

How do I test if Googlebot can crawl a URL?

Enter the URL and choose Googlebot as the crawler in a robots.txt tester, or use the URL Inspection tool in Search Console, which reports whether crawling is allowed by robots.txt for that exact page. Use the tester to confirm the file is reachable and the rule matches the crawler you care about.

Where must the robots.txt file be located?

It must sit at the root of the host, for example example.com/robots.txt. Each subdomain and protocol needs its own file, and a file in a subfolder is not read by crawlers. Subdomains such as blog.example.com need their own separate robots.txt file at their own root.

• Robots.txt Generator: Build a valid robots.txt with allow/disallow rules and sitemap directives.
• AI Crawler Access Checker: Check whether your robots.txt allows or blocks Googlebot, GPTBot, ClaudeBot, PerplexityBot and other AI crawlers.
• Indexability Checker: Run all indexing checks on a URL: status, redirects, robots rules, meta robots, headers and canonical.

Browse all 41 free technical seo tools, or see the full free SEO toolbox. If you would rather have this handled for you, take a look at our technical SEO services and the results from real projects.