robots.txt is a plain-text file at the root of a domain (https://example.com/robots.txt) that tells web crawlers which paths they are allowed to visit. It is part of the Robots Exclusion Protocol (RFC 9309). Googlebot, Bingbot, and all 22 tracked AI crawlers read robots.txt before making requests.

Why add AI-specific rules to robots.txt?

Starting in 2025, two classes of AI crawler emerged: (1) training crawlers that copy your content into model corpora and (2) citation crawlers that fetch your page to answer a user question and cite you back. Most brands want to block class 1 and explicitly allow class 2. Our generator writes those rules for you automatically.

Which crawlers are blocked by the AI-aware rules?

Training-only user agents: CCBot (Common Crawl, widely used as training corpus), cohere-training-data-crawler (Cohere training), Applebot-Extended (Apple Intelligence training), and Google-Extended (Google Gemini training). Citation crawlers remain explicitly allowed.

Which crawlers are explicitly allowed?

Citation and user-driven crawlers: ClaudeBot (Anthropic search), ChatGPT-User (OpenAI user-initiated Browse), Perplexity-User (Perplexity pro browse), and GoogleOther (Google non-training extended crawler). The standard wildcard (*) block still applies first, so your existing Allow/Disallow paths are respected.

How do I validate my robots.txt after deploying?

Use Google's robots.txt Tester in Search Console, then run the Aevra AI Crawler Audit, which simulates all 22 tracked AI crawlers, highlights any accidental full-site blocks, and flags missing explicit rules that would improve transparency and citation rate.

Is the generator free?

Yes. Unlimited use, no sign-up, no data sent to our servers. Everything runs in your browser.

Classic SEO tool · AI-crawler aware · 18+ UA rules

Free robots.txt Generator

Block training crawlers. Allow citation crawlers. Sitemap URL, crawl-delay, Allow/Disallow paths. All copy-paste ready.

Set paths & sitemap

Allow public content, block private/admin paths.

Toggle AI rules

One click = 18+ explicit AI crawler rules.

Copy & upload

Save to /robots.txt on your domain root.

Sitemap URL

Allow paths (one per line)

Disallow paths (one per line)

Crawl-delay (seconds) (leave blank to omit)

Enable AI crawler rules (recommended)

Automatically adds per-UA rules for 18+ AI crawlers:

🚫 Block (training):

CCBot, Applebot-Extended…

✅ Allow (citation):

ClaudeBot, ChatGPT-User…

Included AI crawler groups:

BLOCKCCBot— Common Crawl (training corpus)

BLOCKcohere-training-data-crawler— Cohere (training)

BLOCKApplebot-Extended— Apple (training)

BLOCKGoogle-Extended— Google (training)

ALLOWClaudeBot— Anthropic (citation search)

ALLOWChatGPT-User— OpenAI (user-initiated browse)

ALLOWPerplexity-User— Perplexity (user-initiated)

ALLOWGoogleOther— Google (non-training extended)

robots.txt Preview

# robots.txt — generated by Aevra AI Robots Generator
# Generated: 2026-09-11

User-agent: *
Allow: /
Allow: /docs
Allow: /blog
Disallow: /admin/
Disallow: /api/
Disallow: /dashboard/
Disallow: /private/

# ============================================================
# AI crawler rules — block training-only crawlers,
# explicitly allow citation crawlers for AI search visibility.
# ============================================================

# ── Block training-only crawlers ──────────────────────────
# Vendor: Common Crawl (training corpus)
User-agent: CCBot
Disallow: /

# Vendor: Cohere (training)
User-agent: cohere-training-data-crawler
Disallow: /

# Vendor: Apple (training)
User-agent: Applebot-Extended
Disallow: /

# Vendor: Google (training)
User-agent: Google-Extended
Disallow: /

# ── Allow citation crawlers ───────────────────────────────
# Vendor: Anthropic (citation search)
User-agent: ClaudeBot
Allow: /

# Vendor: OpenAI (user-initiated browse)
User-agent: ChatGPT-User
Allow: /

# Vendor: Perplexity (user-initiated)
User-agent: Perplexity-User
Allow: /

# Vendor: Google (non-training extended)
User-agent: GoogleOther
Allow: /

Sitemap: https://example.com/sitemap.xml

Why this generator is different

18+ AI crawlers covered

Most generators only write a User-agent: * block. We write explicit per-UA rules for every major training and citation crawler so you control access at the platform level.

Smart block vs allow

One click splits crawlers into the two classes that actually matter: training crawlers (blocked by default) and citation crawlers (allowed by default). Tuned per 2026 vendor docs.

Valid syntax

Every line follows RFC 9309 format. No typos, no missing blank lines between groups, no trailing whitespace that confuses parsers.

Need an AI crawler audit for your live site?

Aevra simulates 22 tracked AI crawlers against every URL, flags accidental blocks on high-priority pages, compares your stance to your competitors, and estimates the AI-traffic impact of each rule change.

Unlock full audit Go to dashboard →