SEO & Website Audit
Robots.txt Tester
Robots.txt precedence rules — longest match wins, allow can override disallow — are easy to get backwards by hand. This evaluates them exactly as crawlers do.
FreeNo sign-upLive data
Allowedno matching rule — default allow
Sitemaps declared: https://example.com/sitemap.xml
How it works
- 1Paste robots.txt content, or fetch it live from a domain
- 2Enter the path you want to test
- 3Pick a user-agent and see whether it's allowed, and which rule matched
Who this is for
- Debugging why a page won't get crawled
- Confirming a new disallow rule doesn't block more than intended
- Checking rules for a specific bot like GPTBot separately from the default group
Frequently asked questions
- Which user-agent group does a bot use if it's not listed?
- Crawlers fall back to the User-agent: * group if no rule set specifically names them.
More tools
More in SEO & Website Audit
Sitemap Finder
Probes a domain's common sitemap paths and reports which ones exist.
SEO & Website Audit
Sitemap Validator
Checks a sitemap against the sitemaps.org schema and flags malformed entries.
SEO & Website Audit
Sitemap Link Counter
Counts URL and nested-sitemap entries in a sitemap file.
SEO & Website Audit
Sitemap URL Extractor
Extracts every URL from a sitemap into a plain, copyable list.
SEO & Website Audit
XML Sitemap Generator
Builds a valid sitemap.xml from a plain list of URLs.
SEO & Website Audit
Sitemap Index Generator
Builds a sitemap index file referencing multiple child sitemaps.
SEO & Website Audit
Ask AI about Internet Compass