AI-Driven Proof Generation Marks a Shift in Software Assurance Capabilities

Anthropic is leveraging its Claude model to mathematically verify the correctness of existing software code. This initiative represents an effort to address a critical issue regarding AI-generated software, though significant practical limitations currently exist.

The process of formally verifying that software operates without error has historically been prohibitively expensive. For example, a team of specialists required 20 person-years to verify the seL4 operating system kernel, a task that limited the method to high-stakes environments such as Airbus avionics software. However, Anthropic's Frontier Red Team, alongside partner Theorem, delegated this proof work to Claude. In a few weeks, the model generated approximately 180 million lines of mathematical proof code using test programs from 123 open-source libraries within Google's OSS-Fuzz security project.

Claude does not write new code but instead verifies existing software. It achieves this by analyzing compiled machine code and transforming crash tests into mathematical problems. An independent program, Rocq, automatically verifies the proofs. While the researchers describe the process as 'fully autonomous' proof generation, it is noted that the tools and formal foundations were developed by humans at Theorem.

During the generation of proofs, Claude identified hundreds of bugs and fixed them. Many of these issues were invisible to conventional testing because they only emerge under specific conditions, such as unusually large input files. In one instance involving a PDF reader, the team discovered as many security bugs in a week as are typically reported for the software over a year. Another instance involved finding a logical error that remained undetected for over a decade.

Anthropic notes that 'formally verified' does not guarantee 'error-free.' Proofs are valid only under strict conditions, including a specific processor architecture, single-threaded programs, and inputs under 64 MB, whereas real-world libraries operate on various hardware and configurations. Additionally, compute costs are high, with individual proofs requiring hundreds of millions of tokens. Anthropic does not report total project costs. However, the company believes that if AI verifies its own code, developers can trust AI-generated software more and spend less time reviewing it. This reduces costs not only for verification but for the entire coding process. A smaller test run also indicates that proofs are reusable, and newer models are becoming more efficient: Opus 5.5 solves half the proofs with about 40 percent fewer tokens than Opus 5.

The project strengthens the case for software automation. If AI can write code and guarantee its correctness under defined conditions, the question shifts from 'Can AI program?' to 'Can AI prove that the code it writes is safe?' If the answer is yes, the consequences could go well beyond today's coding tools. Anthropic expects formal verification with AI will soon become as practical as standard testing, and Theorem's roadmap already targets critical infrastructure like glibc, OpenSSL, and cURL.