AI tools are excellent at finding bugs inside a code diff. They are far less reliable at catching the ones that live outside it. A recent case study makes this painfully clear: Copilot, armed with rules, caught all eight bugs that were part of a code change. It missed both bugs that were not. The result was an overbilling of 11 days for a customer, caused by a function that looked correct at first glance. The guard on negative enrollment days was fine. The function it called was not. An invariant test would have found it. The AI did not.
This is not a failure of the technology. It is a failure of our assumptions about what AI can do. We are seeing a parallel pattern in hardware engineering, where Nominal's valuation triples to $3B as AI reshapes hardware engineering. The promise is powerful, but the risk is the same: AI optimises what it can see. It does not automatically consider what it cannot. The lesson from the Copilot case is that AI is a superb assistant for the narrow task of reviewing a diff. It is not a replacement for the broader thinking that catches logic errors spanning multiple functions or states. The missed bugs were not subtle syntax errors. They were logical gaps in how the system behaved across different code paths. An AI that only looks at the changed lines will never see them.
This reinforces a point made in our recent piece on creativity: Let AI Polish Your Ideas, Not Dream Them Up. The same principle applies to code review. AI can polish your diff. It can catch the typos, the misplaced brackets, the obvious oversights. It cannot dream up the invariant test that checks whether the system is billing correctly across all states. That requires a human to step back and ask: what else could go wrong? The AI will not ask that question. It does not know what it does not know.
The practical takeaway is direct: do not let AI lull you into a false sense of completeness. If you use Copilot or similar tools to review code, treat their output as a first pass, not a final verdict. The two missed bugs in this case were invisible to the diff review because they were not in the diff. They were in the assumptions the code made about the data it was processing. An invariant test would have caught them because invariant tests are designed to check those assumptions. That is the kind of testing AI cannot automate until it learns to question the problem itself, not just the patch.
The open question is whether we will learn this lesson before the next overbilling hits a customer. The tools are getting better, but the gaps they leave are not shrinking. They are simply moving to places we are not looking.