Is your website blocking AI assistants without knowing it?

Somewhere in the last two years, “can ChatGPT find us” became a question business owners ask out loud. The usual advice sends you to your robots.txt to check. We measured whether that advice works, and it mostly does not, because the block that hides a company from AI assistants is almost never in the file people are told to look at.

This is one finding out of the Baltic Web Audit, our near-census of the Latvian, Estonian and Lithuanian web in August 2026.

In short

Yes, quite possibly, and your robots.txt will not tell you. We asked live servers directly rather than reading their published rules, and 7% of Latvian sites, 5% of Lithuanian and 3% of Estonian refused the AI search crawler outright. The same servers refused a plain research bot at under half a percent, so the refusal is aimed at AI specifically rather than at automation. And it is mostly not a decision anyone made: among Baltic sites that publish a robots.txt at all, more than nine in ten have never written a single line about AI crawlers.

What a server does is not what a file says

For each site that answered our ordinary request, we sent the same request again three more times: once identifying as an AI assistant’s crawler, the kind that fetches a page so an answer engine can cite it, once as an AI training crawler, and once as a plain research bot. The crawler always said honestly who it was, because being refused is itself one of the measurements.

That is the whole trick, and it is why the numbers look different from every previous survey of this subject. A robots.txt file records what a site asks crawlers to do. It has no power to stop anything. What we counted is what the server actually did when asked.

The two layers disagree badly. A robots.txt exists at all on 61.8% of live Latvian sites, 60.1% of Estonian and 68.1% of Lithuanian. Of the sites that have one, rules naming an AI crawler appear on 9.5% in Latvia, 5.6% in Estonia and 6.0% in Lithuania. So the declared-policy layer is thin, and the enforced layer underneath it is several times thicker.

There is one more number worth putting beside the 7%. AI training crawlers are refused far more often: 15% in Latvia, 18% in Lithuania, 8% in Estonia. The Baltic web clearly distinguishes between being read and being learned from, and it dislikes the second a great deal more. That distinction is a reasonable one to hold. The problem is that a large number of sites are enforcing the first without meaning to.

The crawlers are not one thing, and the vendors say so

This matters because “block the AI bots” was widely repeated advice in 2023 and 2024, when most people had heard of exactly one bot.

Both OpenAI and Anthropic publish the split in their own documentation, and both make each setting independent:

Vendor	Search, the one that puts you in the answer	Training	User-triggered fetch
OpenAI	OAI-SearchBot	GPTBot	ChatGPT-User
Anthropic	Claude-SearchBot	ClaudeBot	Claude-User

OpenAI’s own wording is unambiguous about the cost of the first column: sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links”. Anthropic says the same of Claude-User, that disabling it “prevents our system from retrieving your content in response to a user query”. Both documents were checked live on 28 August 2026.

So there are three separate decisions here wearing one name. Refusing the training crawler is a defensible position about whether your writing trains somebody’s model. Refusing the search crawler removes you from a growing share of the questions your customers ask, and it buys you nothing in return.

We wrote up which crawler does what, and how to read your own file, in what ChatGPT can see about your business. The audit adds the part that piece could not know: the file is only the top layer, and the enforced layer underneath it is a great deal thicker.

Nobody chose this. A vendor chose it.

The single most useful thing to understand about the 7% is that, for most of those sites, it is probably not a decision anyone at the company ever made.

On 1 July 2025, in a post it titled Content Independence Day, Cloudflare announced that it was “changing the default to block AI crawlers unless they pay creators for content”. Cloudflare was detected on 12,950 sites in Latvia, 12,220 in Estonia and 17,769 in Lithuania in this study, and those are detection floors rather than exact counts.

That is the mechanism. A café, a law firm or a manufacturer signs up for a CDN because their site was slow or getting hammered by bots, inherits a crawler policy in the process, and becomes invisible to the assistant their customers now ask for a recommendation. No email, no dashboard warning, no ranking change. This is the shape of nearly everything in our audit: the measured behaviour of a national web turns out to be the default settings of a handful of software vendors, not thousands of individual choices.

One sector actually decided, and it is the one you would guess

Declared policy tells the other half of the story, and there the pattern is completely different. In robots.txt, where a site writes down what it wants rather than letting a product enforce something, one sector stands alone: 21% of Latvian media and publishing sites declare an AI search block, against 9% in Estonia and 11% in Lithuania, while every other sector sits between 1 and 12%.

Publishers understand precisely what an answer engine does to a page view, and they have written their objection down. Everyone else is mostly carrying a default they never chose. If your industry is not publishing, and your site refuses AI search crawlers, that is worth ten minutes of somebody’s attention.

Check your own site in five minutes

Three checks, in order of how much they tell you.

One, read the file. Open https://yourdomain.com/robots.txt in a browser and search it for OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot and the wildcard User-agent: * block. This is the only layer most people ever check, and on the evidence above it is the least likely to hold the answer.

Two, ask your server the same question twice. In a terminal, compare an ordinary browser request with one that identifies as the AI search crawler:

curl -s -o /dev/null -w "browser: %{http_code}\n" https://yourdomain.com/
curl -s -o /dev/null -w "ai search: %{http_code}\n" \
  -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" \
  https://yourdomain.com/

Two 200s and you are readable. A 200 followed by a 403, 429 or a challenge page and something in your stack is deciding crawler policy for you.

One honest caveat on that test, because it is the first thing a developer will ask. A user agent string is only a claim, and bot-management products also verify the real crawler by IP address, so a refusal here is not proof that OpenAI itself is refused. What it does prove is that a policy exists, that it fires on this class of request, and that you did not know about it. That is enough to go looking.

Three, look at the product that is doing it. If you are on Cloudflare, the relevant panel is called AI Crawl Control, formerly AI Audit, and it shows which AI services have been reaching your content and lets you set access per crawler. The same setting exists under different names in other CDN and WAF products. Verified live on 28 August 2026.

What you can fix yourself, and what needs a developer

You can do this alone if the block is declared or dashboard-level: editing the robots.txt lines, or flipping the per-crawler setting in your CDN panel. Both are reversible in a minute, and both should be a deliberate choice per crawler rather than a blanket allow or deny.

You need a developer when the two checks disagree and the panel does not explain why. That usually means a firewall rule, a rate limiter, a security plugin or a reverse proxy is matching on user agent somewhere in the chain, and finding which one requires reading logs rather than guessing. It is an hour of work, not a project, but it is not a settings toggle either.

Readable is not the same as quotable

Being allowed in is the floor, not the goal. A site that answers every AI crawler politely and then says nothing worth extracting will not be cited either, and that is a separate problem with separate causes, which we set out in why AI assistants cite everyone except you.

There is also a third failure mode in our data, arriving from the opposite direction: a small share of sites ship an almost empty HTML file and build the entire page with JavaScript. To a person those sites look fine. To anything reading the page without running a full browser, which includes most AI crawlers, they are close to blank. Blocking is not the only way to be unreadable.

Why we can say any of this

The figures here come from a near-census rather than a sample. We measured 293,806 of the 306,872 Baltic domains in our frame across July and August 2026, from one fixed location, with a crawler that identified itself honestly on every request. Everything is published as an aggregate and no individual site is ever named beside a weakness. The method and every chapter is in the Baltic Web Audit.

If you would rather know than guess, send us the address and we will run the same crawler-access checks against your site and tell you what your server is actually doing.