Big Tech Agrees on One robots.txt Rule to Block AI Training

Four of the world's biggest AI labs just agreed on something they rarely agree on: a single, shared way for any website to say "don't train your models on my content." Anthropic, OpenAI, Google and Microsoft all confirmed on October 30 that they'll honor one new signal in robots.txt — the small text file sitting on nearly every website that tells automated bots what they can and can't do.

What actually changed

Until now, if you ran a blog, a news site, or even a small business page, blocking AI companies from scraping your content for model training meant juggling a different rule for every company — one line for OpenAI's crawler, another for Google's, a third for Anthropic's, and so on. Miss one, and your articles could still end up as training data somewhere.

The new approach, laid out in an IETF draft (that's the Internet Engineering Task Force — the group that proposes technical standards for how the internet works, though a "draft" at this stage is a proposal, not yet an official rule everyone must follow) written by Anthropic engineer Kevin Kelley, introduces one token called ai-training. Site owners add it to their existing robots.txt file, and in theory, every participating AI company's crawler reads it the same way.

• It only covers training — using your content to adjust a model's internal parameters. It does not stop a crawler from fetching or indexing your page for something like search or chatbot citations.
• Anthropic says it has already started rolling out support.
• OpenAI says implementation is coming "in the coming months."
• Google and Microsoft are both targeting sometime around early 2027, with Microsoft naming February 1, 2027 as its start date.

"This policy token is not a directive for crawling behavior," the draft itself states — a reminder that opting out of training doesn't make a site invisible to AI tools, it just keeps that content out of the next model's weights.

Why it's not a real opt-out, yet

Here's the catch nobody's hiding: robots.txt has always worked on the honor system. There's no technical lock stopping a crawler from ignoring the file entirely — it relies on companies choosing to respect it, the same way search engines have mostly respected robots.txt for decades out of self-interest and reputation. A proposal with four big names behind it carries weight, but it isn't law, and it isn't retroactive. Content already scraped and folded into an existing model stays there; this only affects what gets collected going forward, and only from companies that actually implement it.

Google has also flagged a wrinkle worth knowing: opting out under the shared token without separately addressing its existing Google-Extended settings could end up blocking your pages from showing up as citations inside Gemini's answers too — not just training. So the fine print matters more than the headline here.

Why this matters for Indian publishers and developers

India doesn't have a law that specifically governs whether AI companies can scrape a website's content to train a model — the DPDP Act (India's Digital Personal Data Protection Act) covers personal data, not published articles, code, or blog posts, which is exactly the gap this voluntary industry signal tries to fill. We've covered what the DPDP Act actually requires of businesses here, and the contrast is telling: that's binding law with penalties, this is four companies agreeing to behave.

For India's huge base of independent publishers, regional-language news sites, and solo developer blogs — many of whom have quietly watched their content show up paraphrased in chatbot answers with no traffic or credit flowing back — this is the first time blocking that behavior takes one line of text instead of a legal notice or a negotiation nobody has leverage to win. The Digital News Publishers Association here has pushed back on AI scraping for over a year; this doesn't resolve that fight, but it hands smaller Indian sites a tool the big players didn't bother building for them specifically — it's just sitting there in an open standard anyone can use.

What to actually do with this

If you run a website and want to use the new signal, the mechanics are simple enough that you don't need a developer to do it for you:

1. Open your site's robots.txt file (usually at yoursite.com/robots.txt).
2. Add a new user-agent group targeting the shared ai-training token, following the format each company publishes in its documentation.
3. Check back in a few months — since Google and Microsoft haven't switched on enforcement yet, adding the rule today is more "getting ahead of it" than an immediate fix.

What's genuinely new here isn't the technology — robots.txt is nearly 30 years old — it's that the AI industry's biggest rivals found one thing to standardize on before regulators forced their hand. Whether that holds once the enforcement deadlines actually arrive is the part worth watching.