OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI across multiple crawlers

Publishers can now prevent their content from being scraped for AI training with a single setting.

Four major AI companies—OpenAI, Google, Microsoft, and Anthropic—have agreed to adopt a new internet standard that will allow publishers to opt out of having their content scraped for AI training. This new standard, proposed by Anthropic and intended for the IETF, will use a "policy token" within a website's robots.txt file. Publishers will no longer need to individually manage opt-out preferences for each AI company, streamlining the process. This move addresses growing concerns among publishers about unauthorized scraping of their content by AI crawlers. OpenAI, Google, and Anthropic will block their AI crawlers from sites displaying this token, while Microsoft will also offer granular controls to opt out of AI training or AI grounding specifically, though websites will still be accessible in Bing search results. Anthropic has already implemented the signal, with Google and Microsoft planning support next year, and OpenAI in the coming months.

This consensus marks a significant shift in how AI companies engage with web content, moving towards a more publisher-friendly ecosystem where explicit consent is prioritized. The adoption of a standardized opt-out mechanism directly addresses the growing tension between AI development's insatiable data needs and content creators' rights to control their intellectual property. It establishes a clear, albeit potentially incomplete, framework for respecting publisher preferences, potentially influencing future discussions on data licensing and AI training.

The broad support for this standard suggests a market-wide acknowledgment of the need for ethical data sourcing and a desire to preempt more stringent regulatory measures. Publishers gain a powerful, unified tool to assert control over their digital assets, which could lead to new models for content monetization or licensing for AI training. The varying implementation timelines and Microsoft's more granular options highlight that the industry is still navigating the nuances of AI data usage, and further refinements are likely.