Does Cloudflare block AI crawlers? What each setting does

On 1 July 2025 Cloudflare announced that it was “changing the default to block AI crawlers unless they pay creators for content”. Since then “Cloudflare blocks AI by default” has been repeated as a fact, and it sends site owners looking for one switch. There is no single switch. Cloudflare has at least four settings that touch AI crawlers, they do different things, and going by its own documentation none of the defaults among them blocks AI search.

This is the how-to that follows from one finding in the Baltic Web Audit, our near-census of the Latvian, Estonian and Lithuanian web. The finding is in is your website blocking AI assistants without knowing it, where we pointed at that default as the likely mechanism. Read closely, the documentation never supported that for search crawlers, and since Cloudflare updated it on 2 November 2026 it describes no blocking default for new domains at all. This piece is about the dashboard.

In short

Going by Cloudflare’s documentation as read on 3 November 2026, Cloudflare does not block AI crawlers by default. Its AI Crawler and Agent access policies page says new domains allow Search, Agents and Training, and the robots.txt it can manage for you adds rules only after someone sets a policy to Block or Disallow. So when a Cloudflare site refuses the crawler that puts pages into ChatGPT’s answers, look for a setting somebody turned on: a policy with Search set to Block, a per-crawler Block in AI Crawl Control, or a custom firewall rule. The first two you can read and change yourself.

What the audit saw, and what it could not see

The audit detected Cloudflare on 12,950 sites in Latvia, 12,220 in Estonia and 17,769 in Lithuania. It also asked every live server the same question several times under different names, and counted the refusals:

Country	Refused an AI training crawler	Refused an AI search crawler
Latvia	15%	7%
Lithuania	18%	5%
Estonia	8%	3%

Training crawlers are refused at least twice as often as search crawlers in every country, and more than three times as often in Lithuania. That gap is the shape you would expect if the settings people switch on target training and leave search alone. Cloudflare’s current documentation does not describe a default that would produce it.

One limit, stated plainly. The audit saw status codes, not dashboards. It can say that a server refused a request. It cannot say which product or which setting produced the refusal on any one site. The rest of this article is the part you can check on your own zone, where the setting is visible.

Three behaviours, set separately

On 1 July 2026 Cloudflare’s changelog announced that “all customers can now manage AI crawlers by behavior” in place of a single Block AI bots toggle. The documentation defines three:

Behaviour	Cloudflare’s definition	Crawlers of this kind
Search	”Collects or indexes your content so it can answer questions about it later.”	OAI-SearchBot, Claude-SearchBot, PerplexityBot
Agent	”Automated activity acting in real time on a person’s behalf to get something done, such as chat fetch bots and browser-use agents.”	ChatGPT-User, Claude-User, Perplexity-User
Training	”Crawls your content to train or fine-tune a model, permanently absorbing your data into the model.”	GPTBot, ClaudeBot, CCBot, Bytespider

A note on the third column. Cloudflare’s bot reference still files crawlers under older category names: AI Search, AI Assistant and AI Crawler. Cloudflare says outright that AI Search is now treated as Search. Placing AI Assistant under Agent and AI Crawler under Training is our reading of definitions that match, AI Assistant being “driven by user action” and AI Crawler collecting content “used for training AI models”. The Crawlers tab in your own dashboard shows the category Cloudflare gives each crawler, and that is the authority.

Each behaviour can be set to Block, Block on pages with ads, or Allow. Training has a fourth option, Disallow, which asks crawlers to stay away through robots.txt instead of refusing them.

Every Cloudflare setting that touches AI crawlers

Setting	Where it is	What it does	Can it refuse AI search?
AI Crawler and Agent access policies	Security, Settings, filter by Bot traffic	Block or allow Search, Agent and Training separately	Yes, if Search is set to Block
Bot Preference Sync (the managed robots.txt)	A switch inside the same policies	Writes your policies into rules placed in front of your robots.txt	No. It asks rather than refuses, and it names search crawlers only if your Search policy blocks them
AI Crawl Control	Its own item in the zone menu, Crawlers tab	Allow or Block per crawler	Yes, per crawler
WAF custom rules	The firewall rules on the zone	Whatever the rule says	Yes

Bot Fight Mode is missing from that table on purpose. It challenges traffic that matches bot patterns, and Cloudflare’s page on verified bots says that “historically, Verified bots have been excluded in default bot configurations across all plans”. A crawler Cloudflare has verified is not its target. It matters for a different reason, which comes up in the test below.

What happened to the 15 September defaults

When we first published this piece on 2 October, the policies page carried this notice: “On September 15, 2026, Cloudflare will set updated defaults for new domains: bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed.” Cloudflare’s bots changelog still announces those defaults in its 1 July 2026 entry.

The page itself now says something else. Updated on 2 November 2026 and renamed AI Crawler and Agent access policies, it reads: “For new domains, Search, Agents, and Training use Allow (do not block) by default.” The notice is gone. We take the page that describes the setting as the current word and the changelog entry as an announcement it has overtaken. Search was allowed under both versions.

Three things follow for a site owner.

A new domain on Cloudflare blocks no AI behaviour by default. If you want training refused, that is now a choice you make in the policy, not something you inherit.

Block on pages with ads is an option, not a default. Cloudflare detects the ads automatically. If someone chose it for Agent, that matters: OpenAI’s bot documentation says that “when users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent.” Refusing that fetch means the assistant cannot open the page a person just asked it about.

Blocking training takes mixed-purpose crawlers with it. The page says that “Training actions also apply to mixed-purpose crawlers used for both Search and Training.” It does not name those crawlers, so we do not either. This is the one place where choosing to block training can cost you a search crawler, and the Crawlers tab is where you would see it.

A caution on dates. Cloudflare changed this page between our first reading on 2 October and our second on 3 November, and its changelog had not caught up when we read it. We report what the documentation said on the day we read it.

The robots.txt Cloudflare writes for you

The managed robots.txt is now driven by the policies. Turn on Enable Bot Preference Sync, set Training to Disallow, and Cloudflare places its own block of rules in front of your file. The example in its documentation disallows eight crawlers: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. It also adds the line Content-signal: search=yes, ai-train=no, use=reference.

No search crawler is on that list. OAI-SearchBot, Claude-SearchBot and PerplexityBot are not disallowed, and neither is Googlebot. Cloudflare adds search crawlers to the file only when the Search policy is set to Block or Block on pages with ads. With every policy set to allow, in its words, “there is nothing to prepend and your robots.txt is served unchanged.”

The same page is direct about what the file can do: “robots.txt compliance is voluntary. The file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level.”

Two smaller points from that page are worth knowing before they surprise you. A domain on the Free plan with no robots.txt of its own, and without the managed file, shows a block of comments called the Content Signals Policy, which defines terms and expresses no preference. And Google Search Console may report “Syntax not understood” for the content-signal line. Cloudflare says it has “observed no impact on crawling rates or SEO” from those reports.

Check your own zone

Four checks. The first three need only a login.

1. Read the AI policies. Security, then Settings, filter by Bot traffic, then AI Crawler and Agent access policies. Note what Search, Agent and Training are each set to, and whether Enable Bot Preference Sync is on.

2. Read your live robots.txt. Open https://yourdomain.com/robots.txt and look for the line # BEGIN Cloudflare Managed content. If it is there, Bot Preference Sync is writing your policies into the file, and everything down to # END Cloudflare Managed Content was written by Cloudflare.

3. Read the Crawlers tab. Open AI Crawl Control. Each crawler has a row with its category, its request count and an Action. Look for Block against any crawler of the Search or Agent kind. Cloudflare’s documentation says a block here “creates or updates a WAF custom rule on your zone”, so it is a block in its own right, separate from the policy in step 1.

4. Ask the server, and read the header. The two-request test in our earlier piece compares a browser with a request that names itself as an AI crawler. On Cloudflare, add -I to see the response headers:

curl -sI -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" \
  https://yourdomain.com/ | grep -i -e "^HTTP" -e "cf-mitigated"

Cloudflare documents that a challenge page always carries the header cf-mitigated: challenge. If you see it, your request was challenged, and here is where Bot Fight Mode comes in: your terminal is not OpenAI, so Cloudflare is looking at an unverified request wearing a crawler’s name. A challenge on this test shows that some protection challenged an unverified request. It does not prove the real crawler is refused. A plain 403 with no challenge header points to a block rule, which is steps 1 and 3.

We ran the same test against this site, which is served through Cloudflare, on 3 November 2026. A browser, OAI-SearchBot, GPTBot and ChatGPT-User all received 200, and our robots.txt carries no managed block.

What to set if your site exists to win customers

Behaviour	Setting	Why
Search	Allow	OpenAI states that sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers”
Agent	Allow	This is a person asking an assistant to open your page
Training	Your choice	It does not decide whether you are quoted. If you block it, check the Crawlers tab afterwards for a search crawler that went with it

If your text is the product, as it is for a publisher, the answer is different, and we set it out in why the media blocks AI.

What you can fix yourself, and what needs a developer

Steps 1 to 3 are yours. They are dashboard settings, each reversible in a minute, and none of them touches the site’s code. On the Free plan AI Crawl Control recognises crawlers by the name they give, which is enough for the well-known ones.

You need a developer when all three look clean and the server still refuses. That leaves a custom rule in the WAF, a rate limit, or something behind Cloudflare such as a security plugin or the host’s own firewall. Finding which one means reading Security, Analytics, Events for the refused request and following it to the rule that fired. It is a short job for someone who reads those logs often.

Why we can say any of this

The country figures come from the Baltic Web Audit: 293,806 of 306,872 Baltic domains measured across July and August 2026, published as aggregates that name no site. Every statement about Cloudflare is from Cloudflare’s public documentation, first read on 2 October 2026 and read again on 3 November 2026, when we corrected this piece because Cloudflare had changed its defaults page. Quotes are given where the wording matters, and it will change again. Check the dates on those pages before relying on ours.

If you would rather have someone read the zone with you, AI search visibility starts with exactly this check: what your server tells each crawler, and whether that is what you meant.