Reviewing AI-Generated Code: Start With the Spec
Why reviewing specifications matters more as AI-generated code scales.
Author:
Certora On the Securing Ethereum in the Age of AI panel at Devcon India on November 3, 2026, the Ethereum Foundation's security team described what happened to its vulnerability inbox in January 2026. It went from about five external reports a week to about 150 a day. Nearly all of it was noise, with 99.95% of the reports describing no real issue. Generating a plausible report had become cheap, and checking one still took a person's time.
Software teams now face the same imbalance with AI-generated code. An agent produces a change in minutes, and a reviewer still has to read it. Mooly Sagiv, Certora's CEO, starts from this problem: once agents write the code, review and testing become the bottleneck. His proposal is to move the human's attention from the code to the specification the code is checked against.
Why is reviewing AI-generated code the bottleneck?
Reviewing AI-generated code is the bottleneck because agents produce changes faster than people can read them, and tests written from the code inherit its assumptions. Reading a change costs a person time in proportion to its size, and agent output has no such ceiling. Reviewing also asks for more than reading. The reviewer has to work out what the change is supposed to do before judging whether it does it, and the code rarely says.
Tests absorb little of that load. Sagiv's experience is that an LLM asked for tests produces many immediately, and most of them restate what the code already does. A test written from the code inherits the code's assumptions, so it seldom reaches the cases where several behaviors interact. Certora's comparison of formal verification, testing and auditing covers what each method catches.
The EF's response shows a way out. It standardized what a report must contain, so that the reporter supplies a proof the EF's triage agent can run itself, such as a Kurtosis setup and a proof of concept. The EF estimates the false positive rate at well below 1% and credits those strict requirements.
A report that arrives with an executable proof costs a machine seconds to check. A code change that arrives with a checkable statement of what it must do can be handled the same way.
How does reviewing a spec change code review?
Reviewing a spec means a person checks the list of requirements a change must satisfy, and a prover checks the code against that list. In spec-driven development the specification is the main artifact the agent builds from, and here a prover also checks the code against it. A specification lists what the code must do in a form a prover can check. What Is Formal Verification? covers the mechanics. The efficiency of spec review as opposed to code review is thanks to two benefits of formal requirements. First, a code diff changes with every commit, while the list of requirements changes only when the product's promises change. Second, a prover can check any version of the code against that list without a person reading it.
Sagiv's claim follows from those two facts. With a complete specification, the code can change freely as long as the specification keeps holding, so reviewing the spec matters more than reviewing the code. Asked whether the programmer's job is now writing and reviewing specifications, he agreed: informal ones first, formal ones in some cases, and sometimes still reading the code, because the LLM cannot be fully trusted.
The code stays in view. The prover checks it against every property in the spec, and a person still reads whatever the spec leaves out. As Certora's cost guide puts it, formal verification gives stronger guarantees only for the properties the spec actually includes.
Example: reviewing an AI-generated USDC vault against its spec
Sagiv's demo used an off-the-shelf coding agent with Certora's formal verification skills added. The prompt asked for an ERC-4626 USDC vault with a 10% performance fee and a 0.5% annual management fee, both paid to a treasury address, plus an owner role that can pause deposits but never withdrawals and can change the treasury address.
Before any code existed, the agent inferred the properties in English:
- No one can withdraw more than their legitimate share.
- User operations cannot cause dilution.
- Each redeem sends 10% of the profit to the treasury.
- Each withdrawal deducts the 0.5% annual fee, prorated per second the tokens sat in the vault.
- Only the owner can pause deposits, and withdrawals always succeed.
- Only the owner can change the treasury address.
This list is the part a human reviews. It reads in a minute, and every line is a promise to users.
Some of these lines say more than the prompt did. The prompt said the owner can pause deposits but never withdrawals, which restricts one role. The agent wrote "withdrawals always succeed", which is a guarantee to every user. An engineer reading the list decides which of the two the vault should make, and that decision happens in English, before any rule exists.
The agent then generated the Solidity vault and translated each property into a rule in CVL, Certora's specification language. The dilution property became this rule:
1rule othersCannotDiluteShareValue(method f, env e, calldataarg args, uint256 x) 2 filtered { f -> isUserAction(f) } 3{ 4 setupVault(e); 5 6 uint256 before = convertToAssets(e, x); 7 f(e, args); // same env => same block 8 uint256 after = convertToAssets(e, x); 9 10 assert after >= before, "no action in the block lowers the value of x shares"; 11}
The rule records what x shares are worth, runs any user action in the same block, and asserts that the value did not fall.
The Prover returned a violation. Because of fee accruals, if Bob mints, Alice's shares are worth less. The agent then explained the counterexample in plain language. The vault stores its high-water mark rounded down, so a deposit, mint or withdrawal can push the rounded share price up by one tick. With a very large share supply, that tick is charged as a performance fee even though holders gained less than one wei. The agent's own estimate was about 5 USDC-wei of dilution on a $100M vault, roughly five millionths of a dollar. The amount is negligible, and the promise that others cannot dilute your shares was still false as written.
The fix raises the high-water mark after every deposit and withdrawal. Fees have just been settled at that point, so any uptick in price is the operation's own rounding and no part of it is profit. The agent applied the fix and reran every rule. The resulting commit, titled "Fix two fee bugs found by formal verification; add specs and report", also stops assets left in an empty vault from being charged as the next depositor's profit, caps a single accrual's fee at half the assets so fee shares cannot explode, and records the component specs, judge verdicts and the final verification report.
The rerun did not finish cleanly. A rule timed out, and a troubleshooting agent proved a supporting theorem in Lean to get it through. The report Sagiv showed at the end still listed one violated property and two timeouts, because he had capped the token budget for the demo.
A reviewer who reads only the diff has to follow fee accrual, a fixed-point high-water mark and share conversion at the same time to find this. A reviewer who reads the spec asks a smaller question: is "user operations cannot cause dilution" a promise this vault should make? Here the answer is yes, and a five-wei leak breaks it. A team could also relax the promise to allow a rounding tolerance. The demo's agent patched the code. Either choice is a decision about what users may rely on, and it takes one sentence to review.
Why is the specification the hardest part of formal verification?
The specification is the hardest part of formal verification because it requires deciding what the system should do, and the code cannot answer that. Formal verification has three challenges: writing the informal specification, translating it into a formal one, and going from the formal spec to a proof or a bug. Sagiv has spent most of his career on the third. He says the first is by far the hardest. Frontier models translate informal specs into formal ones well. Working out what the system should do is where humans still hold the advantage.
Two failure patterns follow. People asked to write a spec often restate the code. Given an assignment that sets x to five, they write "x equals five", which checks the code against itself and adds no security. Similarly, an LLM that writes both the spec and the code can produce code that meets the spec while the two share the same bugs. Sagiv ranks a wrong specification as the worst way a formally verified system fails. Sagiv's blunt summary is that the agent cheats whenever it can, so the report of a run deserves a read.
Certora checks spec quality in two ways. First, it introduces bugs into the code and checks whether any rule catches them. Second, an AI agent evaluates whether each rule actually captures the intended behavior. Sagiv's experience is that verification adds little value when the spec leaves large parts of the system's behavior uncovered. Our article How Formally Verified Code Goes Wrong and How to Catch It covers these failure modes in depth.
What changes for an engineering team?
The spec comes first, and the code stays modular. Certora has worked with Aave since V1, and Sagiv says V4 turned out to be a different system because verification was present from the start and the engineers were already thinking about modularity before Certora joined. The Aave V4 announcement says the collaboration began at the earliest design stages, and the Hub verification report covers the Hub's specification and verification with the Certora Prover. Sagiv also warns that proofs over very complex, tangled code get hard, even with AI.
Reviewers keep their own understanding of the system. On Certora's team, the engineer and the agent each write the rules for the same code and then compare. Each catches bugs and rules the other missed, and security improves over time as long as neither side leans entirely on the other. Certora's write-up of the exercise has more.
How do you start reviewing AI-generated code against a spec?
Before the next AI-generated change merges, ask the agent for the properties the change must satisfy, in English. Have an engineer who did not write the prompt approve or rewrite them, then check the code against them. Teams that want a prover to run that check can use AutoProver, Certora's AI-driven prover. Our AI formal verification cost guide covers pricing, use cases and getting started.
Frequently asked questions
Does spec review replace code review for AI-generated code?
For the properties the spec covers, the spec becomes the main thing a human reviews, and a prover checks the code against it. Behavior outside the spec still needs a human reader.
Who writes the specification when AI writes the code?
An agent can draft it. In the demo, the agent inferred the properties from the prompt, and a person reviews the English version. Sagiv says deciding what the system should do is the step where humans still hold the advantage.
How do I know whether the spec is good enough?
Insert a bug into the code and see whether any rule fails, and measure how much of the code's behavior the rules cover. Certora's article on how formally verified code goes wrong explains the failure modes.
