LLMs are making formal software verification feasible at a much larger scale

Formal verification of software is now 100x faster and cheaper, making provably secure code feasible for widespread use beyond critical infrastructure.

Anthropic has demonstrated that Large Language Models (LLMs) can perform formal software verification at scale, a task previously limited by significant expert human effort. Collaborating with Theorem, Anthropic used its Claude LLM to generate formal proofs of security properties for over 100 real-world open-source software libraries at the RISC-V assembly code level. This effort represents the first large-scale AI application of formal verification to production software at this level. During the process, Claude identified and helped fix bugs that had evaded detection through trillions of test runs. The proofs generated are extensive, with the total Rocq code comprising approximately 180 million lines and individual proofs reaching around 600,000 lines for binaries with a median length of 18,000 RISC-V instructions. These proofs currently assume single-threaded execution on Linux with a fixed memory budget, but Anthropic sees this as a roadmap for future expansions.

The successful application of LLMs to formal software verification signifies a paradigm shift in cybersecurity and software development. Historically, the immense cost and specialized expertise required for formal verification confined its use to highly critical systems like avionics. By automating proof generation and bug discovery, LLMs drastically lower this barrier, potentially enabling widespread adoption of provably secure software across diverse applications. This advancement directly addresses the growing threat landscape where complex exploits are increasingly common, offering a robust defense mechanism that moves beyond traditional testing methods.

This development has significant implications for the software industry, affecting how developers approach security and reliability. The ability to automatically generate formal proofs means that security guarantees, once a luxury, could become a standard feature in software development workflows. Companies that can leverage LLM-driven verification may gain a competitive advantage by delivering demonstrably more secure products, while those that don't risk falling behind in an evolving security landscape. The availability of tools and published proofs also democratizes access to advanced verification techniques, empowering a broader range of developers.