Does Cloudflare block AI crawlers? What each setting does

On 1 July 2025 Cloudflare announced that it was “changing the default to block AI crawlers unless they pay creators for content”. Since then “Cloudflare blocks AI by default” has been repeated as a fact, and it sends site owners looking for one switch. There is no single switch. Cloudflare has at least five settings that touch AI crawlers, they do different things, and going by its own documentation the defaults among them do not block AI search.

This is the how-to that follows from one finding in the Baltic Web Audit, our near-census of the Latvian, Estonian and Lithuanian web. The finding is in is your website blocking AI assistants without knowing it, where we pointed at that default as the likely mechanism. Read closely, the documentation supports that for training crawlers and not for search crawlers. This piece is about the dashboard.

In short

Cloudflare’s documented defaults block AI training crawlers, not AI search crawlers. As the documentation read on 2 October 2026: the older Block AI bots setting covers crawlers that collect content for training, the robots.txt Cloudflare can manage for you disallows eight training crawlers and no search crawler, and the defaults described for new domains from 15 September 2026 block Training and Agent traffic only on pages that display ads, with Search left allowed. So when a Cloudflare site refuses the crawler that puts pages into ChatGPT’s answers, look for a setting somebody turned on: an AI bot policy with Search set to Block, a per-crawler Block in AI Crawl Control, or a custom firewall rule. The first two you can read and change yourself.

What the audit saw, and what it could not see

The audit detected Cloudflare on 12,950 sites in Latvia, 12,220 in Estonia and 17,769 in Lithuania. It also asked every live server the same question several times under different names, and counted the refusals:

Country	Refused an AI training crawler	Refused an AI search crawler
Latvia	15%	7%
Lithuania	18%	5%
Estonia	8%	3%

Training crawlers are refused at least twice as often as search crawlers in every country, and more than three times as often in Lithuania. That gap is the shape you would expect if the common settings target training and leave search alone, and that is what Cloudflare’s documentation describes.

One limit, stated plainly. The audit saw status codes, not dashboards. It can say that a server refused a request. It cannot say which product or which setting produced the refusal on any one site. The rest of this article is the part you can check on your own zone, where the setting is visible.

Three behaviours, set separately

On 1 July 2026 Cloudflare’s changelog announced that “all customers can now manage AI crawlers by behavior” in place of a single toggle. The documentation defines three:

Behaviour	Cloudflare’s definition	Crawlers of this kind
Search	”Collects or indexes your content so it can answer questions about it later.”	OAI-SearchBot, Claude-SearchBot, PerplexityBot
Agent	”Automated activity acting in real time on a person’s behalf to get something done, such as chat fetch bots and browser-use agents.”	ChatGPT-User, Claude-User, Perplexity-User
Training	”Crawls your content to train or fine-tune a model, permanently absorbing your data into the model.”	GPTBot, ClaudeBot, CCBot, Bytespider

A note on the third column. Cloudflare’s bot reference still files crawlers under older category names: AI Search, AI Assistant and AI Crawler. Cloudflare says outright that AI Search is now treated as Search. Placing AI Assistant under Agent and AI Crawler under Training is our reading of definitions that match, AI Assistant being “driven by user action” and AI Crawler collecting content “used for training AI models”. The Crawlers tab in your own dashboard shows the category Cloudflare gives each crawler, and that is the authority.

Each behaviour takes one of three options: block on all pages, block on pages with ads, or allow.

Every Cloudflare setting that touches AI crawlers

Setting	Where it is	What it does	Can it refuse AI search?
Configure AI bot policies	Security, Settings	Block or allow Search, Agent and Training separately	Yes, if Search is set to Block
Block AI bots (older setting)	Security, Settings	Blocks verified bots classified as crawling for AI training, and unverified bots that behave similarly	Only a crawler Cloudflare counts as both search and training
Managed robots.txt	Security, Settings, filter by Bot traffic: “Set your preference to block training in robots.txt”	Puts Cloudflare’s rules in front of your robots.txt	No. It names training crawlers, and it is a request
AI Crawl Control	Its own item in the zone menu, Crawlers tab	Allow or Block per crawler	Yes, per crawler
WAF custom rules	The firewall rules on the zone	Whatever the rule says	Yes

Bot Fight Mode is missing from that table on purpose. It challenges traffic that matches bot patterns, and Cloudflare’s page on verified bots says that “historically, Verified bots have been excluded in default bot configurations across all plans”. A crawler Cloudflare has verified is not its target. It matters for a different reason, which comes up in the test below.

What Cloudflare says changed on 15 September 2026

The Block AI bots page carries this notice: “On September 15, 2026, Cloudflare will set updated defaults for new domains: bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed.”

Three things follow for a site owner.

It is written for new domains. The notice says nothing about changing zones already on Cloudflare. It does add that before that date “all customers can opt out of these new defaults”, so an existing account is worth a look as well.

It applies to pages that display ads, which Cloudflare detects automatically. A company site with no advertising on it is outside the default as written. An ad-funded blog on a new domain is inside it, and there the Agent half matters: OpenAI’s bot documentation says that “when users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent.” Refusing that fetch means the assistant cannot open the page a person just asked it about.

Blocking training now takes mixed-purpose crawlers with it. The same notice says crawlers “that combine Search and Training will also be blocked by all configurations to block AI training”, the older Block AI bots option included. The page does not name those crawlers, so we do not either. This is the one place where choosing to block training can cost you a search crawler, and the Crawlers tab is where you would see it.

A caution on dates. That page was last updated on 1 July 2026 and still describes 15 September in the future tense, and Cloudflare’s bots changelog had no entry confirming the switch when we read it on 2 October. We are reporting what Cloudflare documented, not something we watched happen.

The robots.txt Cloudflare writes for you

If “Set your preference to block training in robots.txt” is on, Cloudflare places its own block of rules in front of your file. The example in its documentation disallows eight crawlers: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. It also adds the line Content-signal: search=yes, ai-train=no, use=reference.

No search crawler is on that list. OAI-SearchBot, Claude-SearchBot and PerplexityBot are not disallowed, and neither is Googlebot.

The same page is direct about what the file can do: “robots.txt compliance is voluntary. The file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level.”

Two smaller points from that page are worth knowing before they surprise you. A domain on the Free plan with no robots.txt of its own shows a block of comments called the Content Signals Policy, which defines terms and expresses no preference. And Google Search Console may report “Syntax not understood” for the content-signal line. Cloudflare says it has “observed no impact on crawling rates or SEO” from those reports.

Check your own zone

Four checks. The first three need only a login.

1. Read the AI bot policies. Security, then Settings, then Configure AI bot policies. Note what Search, Agent and Training are each set to. On an older setup you may see Block AI bots in its place.

2. Read your live robots.txt. Open https://yourdomain.com/robots.txt and look for the line # BEGIN Cloudflare Managed content. If it is there, the managed file is on, and everything down to # END Cloudflare Managed Content was written by Cloudflare.

3. Read the Crawlers tab. Open AI Crawl Control. Each crawler has a row with its category, its request count and an Action. Look for Block against any crawler of the Search or Agent kind. Cloudflare’s documentation says a block here “creates or updates a WAF custom rule on your zone”, so it is a block in its own right, separate from the policy in step 1.

4. Ask the server, and read the header. The two-request test in our earlier piece compares a browser with a request that names itself as an AI crawler. On Cloudflare, add -I to see the response headers:

curl -sI -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" \
  https://yourdomain.com/ | grep -i -e "^HTTP" -e "cf-mitigated"

Cloudflare documents that a challenge page always carries the header cf-mitigated: challenge. If you see it, your request was challenged, and here is where Bot Fight Mode comes in: your terminal is not OpenAI, so Cloudflare is looking at an unverified request wearing a crawler’s name. A challenge on this test shows that some protection challenged an unverified request. It does not prove the real crawler is refused. A plain 403 with no challenge header points to a block rule, which is steps 1 and 3.

We ran the same test against this site, which is served through Cloudflare, on 2 October 2026. A browser, OAI-SearchBot, GPTBot and ChatGPT-User all received 200, and our robots.txt carries no managed block.

What to set if your site exists to win customers

Behaviour	Setting	Why
Search	Allow	OpenAI states that sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers”
Agent	Allow	This is a person asking an assistant to open your page
Training	Your choice	It does not decide whether you are quoted. If you block it, check the Crawlers tab afterwards for a search crawler that went with it

If your text is the product, as it is for a publisher, the answer is different, and we set it out in why the media blocks AI.

What you can fix yourself, and what needs a developer

Steps 1 to 3 are yours. They are dashboard settings, each reversible in a minute, and none of them touches the site’s code. On the Free plan AI Crawl Control recognises crawlers by the name they give, which is enough for the well-known ones.

You need a developer when all three look clean and the server still refuses. That leaves a custom rule in the WAF, a rate limit, or something behind Cloudflare such as a security plugin or the host’s own firewall. Finding which one means reading Security, Analytics, Events for the refused request and following it to the rule that fired. It is a short job for someone who reads those logs often.

Why we can say any of this

The country figures come from the Baltic Web Audit: 293,806 of 306,872 Baltic domains measured across July and August 2026, published as aggregates that name no site. Every statement about Cloudflare is from Cloudflare’s public documentation as it read on 2 October 2026, quoted where the wording matters, and it will change. Check the dates on those pages before relying on ours.

If you would rather have someone read the zone with you, AI search visibility starts with exactly this check: what your server tells each crawler, and whether that is what you meant.