A shared AI training opt-out would make robots.txt matter again

TL;DR: A shared robots.txt opt-out for AI training could reduce crawler policy chaos, but site owners still need to decide what they want indexed, trained on, quoted, and monetized.

What would a shared robots.txt AI opt-out actually change?

Search Engine Journal’s “Google, Microsoft & OpenAI Plan To Honor Shared AI Training Opt-Out” by Matt G. Southern reports that Google, Microsoft, and OpenAI plan to honor a shared robots.txt line that would let sites opt out of AI training across several providers at once.

That matters because the current setup is messy. Publishers are already dealing with multiple crawlers, multiple user agents, multiple policies, and a blurry line between search indexing, answer generation, model training, summarization, retrieval, and citation. One shared instruction would not solve all of that. But it would give site owners a cleaner control surface.

Robots.txt has always been a strange little trust protocol. It is public. It is voluntary. It depends on the crawler choosing to comply. It does not stop scraping by bad actors. It does not prove that a model forgot anything it already trained on. Still, for legitimate companies, it creates a norm. Norms matter when they become operational defaults.

The key phrase in Southern’s report is that companies are “handling it differently.” That is the part to watch. If Google, Microsoft, and OpenAI all say they respect the same line, but define training, retrieval, snippets, AI answers, and product use differently, publishers may get less clarity than the headline suggests.

Where does this get messy for publishers and SEOs?

The first mistake is treating “AI training opt-out” as one decision.

A news publisher, a SaaS company, a local business, and an affiliate site do not have the same incentives. A publisher may want search visibility but not model training. A software company may want its docs inside AI answers because that reduces support load. A local business may not care about training, but absolutely cares whether assistants can find its address, hours, reviews, and service pages.

I run Lucky Domains, where search visibility still starts with boring questions like what should be crawlable, indexable, canonical, and worth ranking.

This is the same kind of question, with higher stakes. Do you want to be absent from training data if that also reduces future discoverability inside AI interfaces? Nobody can answer that cleanly yet. The distribution layer is changing faster than the norms.

There is also a timing problem. If a site adds an opt-out today, that may affect future training runs by companies that respect it. It does not necessarily affect past training. It does not necessarily affect cached copies, third-party datasets, archived content, syndication partners, or models trained by companies outside the agreement. Southern’s report points to a useful coordination step, not a magic eraser.

What should site owners do now?

Do not wait for a universal standard before making a crawler policy. Also do not panic-edit robots.txt because an AI headline sounds scary.

Start with an inventory. Which parts of the site are strategic content assets? Which are product docs? Which pages exist mainly for customer acquisition? Which pages are thin, private, duplicated, or not meant for machine consumption at all? The answer may differ by directory.

Then separate three questions that people keep mixing together: should this page be indexed by search engines, should this content be used in AI training, and should this content appear in AI-generated answers? Those are related, but they are not the same thing.

For most operators, the practical move is to document your intended policy before changing files. Write down which bots you block, which you allow, and why. Keep a changelog. Test that you did not accidentally block search crawling while trying to block training. Revisit it quarterly, because provider behavior and standards will keep shifting.

Practitioner’s take: treat the shared opt-out as a governance tool, not an SEO hack. If your content is a moat, test a conservative policy on non-critical sections first and monitor crawl logs, rankings, referrals, and AI answer visibility. The catch most people miss is that “AI access” is not one channel anymore. Blocking training may be sensible. Blocking every machine path to your content may quietly cut off future demand.