Does blocking AI crawlers here affect Google or Bing?

No — Googlebot and Bingbot are unaffected. AI-crawler rules only target the specific bots listed, like GPTBot or ClaudeBot.

Can robots.txt actually stop a page from being indexed?

Not reliably. Disallow prevents crawling, but a disallowed URL can still appear in search results if linked from elsewhere. Use a noindex meta tag to guarantee exclusion.

SEO & Website Audit

Robots.txt Generator

Robots.txt syntax is small but unforgiving — one wrong slash can deindex a section of your site. This assembles a correct file from plain inputs, including modern AI-crawler controls.

FreeNo sign-upRuns in your browser

Disallow paths (one per line)

Allow paths (one per line, optional)

Sitemap URL

Crawl-delay in seconds (0 = omit)

AI crawlers

How it works

1List paths to disallow or explicitly allow
2Add your sitemap URL and an optional crawl-delay
3Choose how AI crawlers should be treated

Who this is for

Launching a new site with sane crawl defaults
Blocking AI training crawlers without blocking search indexing
Restricting crawl of faceted-navigation or checkout paths

Frequently asked questions

Does blocking AI crawlers here affect Google or Bing?: No — Googlebot and Bingbot are unaffected. AI-crawler rules only target the specific bots listed, like GPTBot or ClaudeBot.
Can robots.txt actually stop a page from being indexed?: Not reliably. Disallow prevents crawling, but a disallowed URL can still appear in search results if linked from elsewhere. Use a noindex meta tag to guarantee exclusion.

Robots.txt Generator

How it works

Who this is for

Frequently asked questions

More in SEO & Website Audit

Sitemap Finder

Sitemap Validator

Sitemap Link Counter

Sitemap URL Extractor

XML Sitemap Generator

Sitemap Index Generator