Is your website blocking AI assistants without knowing it?

Somewhere in the last two years, “can ChatGPT find us” became a question business owners ask out loud. The usual advice sends you to your robots.txt to check. We measured whether that advice works, and it mostly does not, because the block that hides a company from AI assistants is almost never in the file people are told to look at.

This is one finding out of the Baltic Web Audit, our near-census of the Latvian, Estonian and Lithuanian web in August 2026.

In short

Yes, quite possibly, and your robots.txt will not tell you. We asked live servers directly rather than reading their published rules, and 7% of Latvian sites, 5% of Lithuanian and 3% of Estonian refused the AI search crawler outright. The same servers refused a plain research bot at under half a percent, so the refusal is aimed at AI specifically rather than at automation. And it is mostly not written down anywhere an owner would look: among Baltic sites that publish a robots.txt at all, more than nine in ten have never written a single line about AI crawlers.

What a server does is not what a file says

For each site that answered our ordinary request, we sent the same request again three more times: once identifying as an AI assistant’s crawler, the kind that fetches a page so an answer engine can cite it, once as an AI training crawler, and once as a plain research bot. The crawler always said honestly who it was, because being refused is itself one of the measurements.

That is the whole trick, and it is why the numbers look different from every previous survey of this subject. A robots.txt file records what a site asks crawlers to do. It has no power to stop anything. What we counted is what the server actually did when asked.

The two layers disagree badly. A robots.txt exists at all on 61.8% of live Latvian sites, 60.1% of Estonian and 68.1% of Lithuanian. Of the sites that have one, rules naming an AI crawler appear on 9.5% in Latvia, 5.6% in Estonia and 6.0% in Lithuania. So the declared-policy layer is thin, and the enforced layer underneath it is several times thicker.

There is one more number worth putting beside the 7%. AI training crawlers are refused far more often: 15% in Latvia, 18% in Lithuania, 8% in Estonia. The Baltic web clearly distinguishes between being read and being learned from, and it dislikes the second a great deal more. That distinction is a reasonable one to hold. The problem is the sites that refuse the first as well, and have never written that down anywhere.

The crawlers are not one thing, and the vendors say so

This matters because “block the AI bots” was widely repeated advice in 2023 and 2024, when most people had heard of exactly one bot.

Both OpenAI and Anthropic publish the split in their own documentation, and both make each setting independent:

Vendor	Search, the one that puts you in the answer	Training	User-triggered fetch
OpenAI	OAI-SearchBot	GPTBot	ChatGPT-User
Anthropic	Claude-SearchBot	ClaudeBot	Claude-User

OpenAI’s own wording is unambiguous about the cost of the first column: sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links”. Anthropic says the same of Claude-User, that disabling it “prevents our system from retrieving your content in response to a user query”. Both documents were checked live on 28 August 2026.

So there are three separate decisions here wearing one name. Refusing the training crawler is a defensible position about whether your writing trains somebody’s model. Refusing the search crawler removes you from a growing share of the questions your customers ask, and it buys you nothing in return.

We wrote up which crawler does what, and how to read your own file, in what ChatGPT can see about your business. The audit adds the part that piece could not know: the file is only the top layer, and the enforced layer underneath it is a great deal thicker.

A vendor default explains the training block, not the search block

The single most useful thing to understand about these refusals is that the policy sits in a product’s settings, not in anything the company wrote down.

On 1 July 2025, in a post it titled Content Independence Day, Cloudflare announced that it was “changing the default to block AI crawlers unless they pay creators for content”. Cloudflare was detected on 12,950 sites in Latvia, 12,220 in Estonia and 17,769 in Lithuania in this study, and those are detection floors rather than exact counts.

When we first published this piece we called that default the mechanism behind the 7%. Read closely, it explains only half of the finding, and we have corrected this section. Cloudflare’s own documentation, as it read on 3 October 2026, describes defaults that block crawlers collecting content for AI training and leave AI search allowed: the robots.txt it can manage for you disallows eight training crawlers and no search crawler, and the defaults it describes for new domains keep Search allowed. That fits the 15%, 18% and 8% of sites refusing the training crawler. It does not explain the sites refusing the search crawler. On Cloudflare that takes a setting somebody switched on, and we go through each one in does Cloudflare block AI crawlers.

The owner’s side of it is the same either way. A café, a law firm or a manufacturer signs up for a CDN or a security product because their site was slow or getting hammered by bots, ends up with a crawler policy in the process, and becomes invisible to the assistant their customers now ask for a recommendation. No email, no dashboard warning, no ranking change. Whether that policy arrived as a default or as a switch somebody turned on during setup, our audit cannot say: it saw status codes, not dashboards. For the training figure, at least, this is the shape of nearly everything in the audit: the measured behaviour of a national web turns out to be the default settings of a handful of software vendors, not thousands of individual choices.

One sector actually decided, and it is the one you would guess

Declared policy tells the other half of the story, and there the pattern is completely different. In robots.txt, where a site writes down what it wants rather than letting a product enforce something, one sector stands alone: 21% of Latvian media and publishing sites declare an AI search block, against 9% in Estonia and 11% in Lithuania, while every other sector sits between 1 and 12%.

Publishers understand precisely what an answer engine does to a page view, and they have written their objection down. Everyone else is mostly carrying a setting nobody wrote down. If your industry is not publishing, and your site refuses AI search crawlers, that is worth ten minutes of somebody’s attention.

Check your own site in five minutes

Three checks, in order of how much they tell you.

One, read the file. Open https://yourdomain.com/robots.txt in a browser and search it for OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot and the wildcard User-agent: * block. This is the only layer most people ever check, and on the evidence above it is the least likely to hold the answer.

Two, ask your server the same question twice. In a terminal, compare an ordinary browser request with one that identifies as the AI search crawler:

curl -s -o /dev/null -w "browser: %{http_code}\n" https://yourdomain.com/
curl -s -o /dev/null -w "ai search: %{http_code}\n" \
  -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" \
  https://yourdomain.com/

Two 200s and you are readable. A 200 followed by a 403, 429 or a challenge page and something in your stack is deciding crawler policy for you.

One honest caveat on that test, because it is the first thing a developer will ask. A user agent string is only a claim, and bot-management products also verify the real crawler by IP address, so a refusal here is not proof that OpenAI itself is refused. What it does prove is that a policy exists, that it fires on this class of request, and that you did not know about it. That is enough to go looking.

Three, look at the product that is doing it. If you are on Cloudflare, the relevant panel is called AI Crawl Control, formerly AI Audit, and it shows which AI services have been reaching your content and lets you set access per crawler. The same setting exists under different names in other CDN and WAF products. Verified live on 28 August 2026.

What you can fix yourself, and what needs a developer

You can do this alone if the block is declared or dashboard-level: editing the robots.txt lines, or flipping the per-crawler setting in your CDN panel. Both are reversible in a minute, and both should be a deliberate choice per crawler rather than a blanket allow or deny.

You need a developer when the two checks disagree and the panel does not explain why. That usually means a firewall rule, a rate limiter, a security plugin or a reverse proxy is matching on user agent somewhere in the chain, and finding which one requires reading logs rather than guessing. It is an hour of work, not a project, but it is not a settings toggle either.

Readable is not the same as quotable

Being allowed in is the floor, not the goal. A site that answers every AI crawler politely and then says nothing worth extracting will not be cited either, and that is a separate problem with separate causes, which we set out in why AI assistants cite everyone except you.

There is also a third failure mode in our data, arriving from the opposite direction: a small share of sites ship an almost empty HTML file and build the entire page with JavaScript. To a person those sites look fine. To anything reading the page without running a full browser, which includes most AI crawlers, they are close to blank. Blocking is not the only way to be unreadable.

Why we can say any of this

The figures here come from a near-census rather than a sample. We measured 293,806 of the 306,872 Baltic domains in our frame across July and August 2026, from one fixed location, with a crawler that identified itself honestly on every request. Everything is published as an aggregate and no individual site is ever named beside a weakness. The method and every chapter is in the Baltic Web Audit.

If you would rather know than guess, send us the address and we will run the same crawler-access checks against your site and tell you what your server is actually doing.