Understanding the Impact of LLM Watermarking on AI Agent Behavior

AI agent tool calls may now execute the wrong action even if they appear correct to the user.

Anthropic is implementing an invisible watermark in its future Claude models, leveraging Google DeepMind's SynthID-Text technology. This watermarking is designed to comply with the EU AI Act's Article 50(2), which mandates that AI systems generating synthetic text must mark their outputs as machine-readable and detectable. The implementation of SynthID-Text modifies the LLM's token generation process, a phenomenon termed 'sampling drift.' This drift can affect the model's safety behaviors, such as its refusal of harmful requests, and its susceptibility to prompt injection. Moreover, for AI agents that utilize these LLMs, the sampled tokens can directly influence tool selection and the arguments passed to those tools. Researchers found that this watermarking process results in measurable changes to both model refusal behavior and AI agent tool-calling capabilities. These effects are dependent on the specific model and watermark key used, and aggregate metrics can obscure significant behavioral shifts. A detailed analysis revealed that while net accuracy changes might seem small, a substantial percentage of individual tool-call verdicts differ between watermarked and unwatermarked runs, indicating a meaningful impact on agent functionality.

The core innovation here is the shift from watermarking as a content provenance tool to its integration directly within the generation process, thereby influencing AI agent behavior. This 'sampling drift' means that even non-distortionary watermarks can alter decisions about which tools an agent uses and what data it passes, potentially leading to unintended consequences. The empirical findings highlight that these changes, particularly in tool-call correctness, are significant and widespread across various models, suggesting a systemic challenge for AI agent development and deployment.

This development has direct implications for AI safety and security frameworks, as changes in refusal behavior under prompt injection become more critical when coupled with altered tool-use capabilities. Developers and security professionals need to re-evaluate their robustness testing methodologies, moving beyond aggregate performance metrics to analyze paired disagreement rates. The study indicates that a substantial churn in agent behavior can occur even with minor apparent accuracy shifts, necessitating a deeper understanding of how watermarking keys and model architectures interact to influence agent actions.