AI crawler access

ExpertDuration ~12 min video + 30 min hands-onTools Any site’s /robots.txt in a browser, A robots.txt validator, The Level 3 workbook crawler-audit checklist

Why this lesson

You can write the most citable page on the internet, but if the AI’s crawler is disallowed in your robots.txt, that platform never reads it — and can’t cite what it can’t see. This is a shockingly common own-goal: Ahrefs found around 5.9% of 140 million sites block GPTBot, and many owners never chose to. Rules get inherited from templates, old configs, or a host that flips a switch. Cloudflare’s “manage AI bot traffic” feature is enabled by default and will quietly add blocks for you. So before you optimize anything, you audit access: know the AI crawler taxonomy, read a robots.txt for it, and make a deliberate allow-or-block decision instead of an accidental one.

Watch

Watch for: The first check — auditing robots.txt for GPTBot, OAI-SearchBot, ClaudeBot and Google-Extended — and Sam Oh's warning that Cloudflare's default 'instruct AI bot traffic' feature can block you without your knowledge. Note the honest take on llms.txt (no major provider officially uses it yet) and that robots.txt is the file that actually matters right now.

The explainer

What an AI crawler is (and why there are so many)

An AI crawler is a bot that fetches web pages on behalf of an AI company, and the same company often runs several for different jobs — training the model, powering live search retrieval, and fetching a page a user pasted into a chat. That distinction matters: you might be happy to appear in ChatGPT’s search answers while opting out of model training. Crawlers you don’t recognize will keep multiplying, so the skill isn’t memorizing every name — it’s reading /robots.txt and knowing which user-agents map to which platform and which job.

The taxonomy worth knowing in 2026

Go to any site’s yourdomain.com/robots.txt and look for User-agent lines naming these:

• GPTBot — OpenAI’s crawler for gathering training data. Block it and you opt out of ChatGPT’s training.
• OAI-SearchBot — OpenAI’s crawler that powers ChatGPT Search results. This is the one to keep open if you want ChatGPT to cite you live.
• ChatGPT-User — fetches a specific page when a user acts in ChatGPT (e.g. clicks or asks it to open a link).
• ClaudeBot — Anthropic’s crawler (the bot behind Claude).
• PerplexityBot — Perplexity’s own fast crawler; it indexes and cites within days.
• Google-Extended — a control token, not a bot. It governs Gemini training and grounding only; it does not affect Googlebot or your normal Search ranking. Blocking it keeps you in Search while opting out of Gemini training.
• CCBot (Common Crawl), Applebot-Extended (Apple Intelligence training opt-out), Bytespider (ByteDance/TikTok) — secondary but common.

A trap to remember: because Google-Extended is separate from Googlebot, people block it thinking they’re “blocking AI” and are surprised it changes nothing in Search — that’s by design.

Scoring access: the crawler access score

The GEO toolkit turns this audit into a number. Start at 100 and deduct for what’s blocked: −15 for each critical crawler disallowed (Googlebot, GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot), −5 for each secondary crawler, and −10 if there’s no sitemap declared. A page can be flawlessly citable and still score low here — and a low score caps everything downstream, because access is the gate that comes before content. Run it, and every deduction is a specific, fixable line in a file.

The allow-or-block decision: visibility vs control

This isn’t automatic — it’s a real trade-off. Allow AI crawlers when you want visibility and citations (most businesses whose goal is to be recommended). Block them when you have a genuine reason: protecting proprietary or paywalled content, or a policy stance on your work being used to train models. The key is to decide per crawler and per job. A common, sensible posture: allow the search/retrieval bots (OAI-SearchBot, PerplexityBot, Googlebot) so you get cited, while opting out of pure training crawlers (GPTBot, Google-Extended, Applebot-Extended) if training use concerns you. What you don’t want is the third option — blocking by accident and never knowing. And remember robots.txt is an honor system: reputable AI crawlers obey it, but it’s a request, not a wall.

Don’t confuse this with llms.txt

You’ll hear about llms.txt — a proposed file to summarize your site for AI. As of 2026 no major provider officially uses it (covered honestly in a later lesson), so it isn’t a substitute here. robots.txt is the file that actually controls AI access today. Fix access first.

Do it

1. Open yourdomain.com/robots.txt for a real site — your own, or a business you want to study.
2. Scan every User-agent block for the taxonomy above: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Applebot-Extended, Bytespider.
3. For each, record: allowed or disallowed? Mark any critical crawler that’s blocked in red.
4. Compute the crawler access score: start at 100, −15 per critical crawler blocked, −5 per secondary, −10 if no Sitemap: line is present.
5. Flag likely accidental blocks — especially if the site is on Cloudflare (check for the default AI-bot rule) or uses an off-the-shelf CMS template.
6. Write the deliberate decision: for this site, which crawlers should be allowed vs blocked, and why (visibility vs control). Note the one-line robots.txt fix for each accidental block. Log it in the Level 3 workbook.

Level 3 workbook — AI crawler audit checklist & access-score worksheetlevel-3-workbook.pdf113 KBOriginal course material — free to use

Terms introduced

Terms introduced in this lesson

• AI crawler
• GPTBot
• OAI-SearchBot
• ClaudeBot
• PerplexityBot
• Google-Extended
• Crawler access score

Each term links to the glossary, where it is defined by the way you just used it.

Check

Check yourself

1. What does blocking Google-Extended in robots.txt actually do?

   It removes you from Google Search entirelyIt blocks ChatGPTIt speeds up your siteIt opts your content out of Gemini model training and grounding — but it does NOT affect Googlebot or your normal Search ranking

2. A site is on Cloudflare and its owner never touched robots.txt, yet GPTBot is disallowed. Most likely cause?

   Google added itGPTBot doesn’t existThe site was hackedCloudflare’s "manage AI bot traffic via robots.txt" feature is on by default and inserted the block automatically — a very common accidental block

3. Why is auditing robots.txt for AI crawlers worth doing before any citability work?

   Because it changes your Google rankingBecause robots.txt controls page speedIt isn’t — crawlers ignore robots.txtIf a crawler like GPTBot is disallowed, the engine can’t read your page at all, so even perfectly citable content is invisible to that platform

Check answers

You can move on when you can… read a real robots.txt, identify which AI crawlers are allowed or blocked, tell Google-Extended apart from Googlebot, compute a crawler access score, and make a deliberate allow-or-block decision for each crawler and job.

Mark this lesson complete

Go deeper

• OpenAI — “GPTBot” docs and Google — “Google-Extended” in Search Central: the first-party specs for what each user-agent controls. Read the source, not a summary.
• Dark Visitors / the community AI-crawler lists: a maintained, current directory of AI user-agents and the companies behind them.
• Next: 3.4 · Entities, schema & sameAs — once crawlers can reach you, help them understand who you are.