A 19th-century fisheries trick is now one of the most practical ways to measure AI recall, and that is a genuinely useful insight for anyone building or buying AI agents. The method, capture-recapture, was tested on 294 CUAD contracts and landed within two points of true recall for most agents using just a 10% hand audit. This is not a theoretical curiosity. It is a concrete, low-cost tool that turns an opaque problem, how many relevant documents did my AI miss?, into something you can actually estimate without reviewing every single result. We have argued before that Let AI Polish Your Ideas, Not Dream Them Up is a useful discipline, and this method extends that same thinking into evaluation: use human effort where it counts, and let the math do the rest.
The elegance of capture-recapture is that it was designed for counting fish in a lake, not documents in a contract review. You catch a sample, tag it, release it, catch another sample, and see how many tags reappear. The ratio tells you the total population. Applied to AI recall, the first "catch" is the set of documents an agent retrieves, the second is a small hand-audited sample, and the overlap reveals the agent's completeness. The results from the CUAD test are striking: with only 10% of documents manually reviewed, the estimate came within two points of the true recall for most agents. That is a level of accuracy that makes the method immediately useful for quality assurance, vendor evaluation, or internal benchmarking. It is also a reminder that How smarter batching in vLLM delivers up to 2.91× faster inference is not the only path to better AI systems, sometimes the gains come from better measurement, not faster computation.
What this means in practice is that you no longer have to guess whether your AI agent is missing a quarter of the relevant contracts or just a few. A 10% audit is not nothing, but it is dramatically cheaper than a 100% review, and the estimate is reliable enough to act on. For teams deploying AI in legal, compliance, or due diligence workflows, this changes the risk calculation. You can now say, with reasonable confidence, that your agent's recall is 92% or 88%, rather than shrugging and hoping for the best. It also opens a direct comparison to the kind of measurement gap we saw in Quantile Regression Reveals a Home Value Ordinary Models Miss by $55K, where a single-point estimate hid a large error. Capture-recapture does not hide the error; it quantifies it.
The specific takeaway is this: if you are running any AI agent that retrieves or classifies documents, you should test capture-recapture on your own data this quarter. The method is simple enough to implement in a spreadsheet, and the 10% audit threshold is actionable. The open question is how well it generalises beyond contracts, to medical records, email archives, or customer support tickets, but the CUAD results are a strong enough signal to warrant exploration. That is the kind of practical, human-centered progress that makes AI tools more trustworthy, not just faster.