Clairic

How Smart Routing and Confidence Scores Let AI Handle the Easy Queries First

A 0.9 confidence cutoff lets Cloudflare's System One model handle the easy queries, routing only the tougher cases to a larger, slower LLM like Luna. This isn't about replacing human judgment; it's about using AI where…

3 min readTowards Data Science

Smart routing is the most practical idea to hit the AI conversation in months, and Cloudflare's System One model proves why. By setting a 0.9 confidence cutoff, System One handles the easy queries on its own and passes the rest to a larger, slower LLM like Luna. Clairic reported 98% accuracy on 72.8% of questions using this approach. That is not a lab result, it is a production metric that tells us something important: most of the work does not need the heavy artillery. The insight changes how we think about building AI systems that actually scale.

For anyone who has watched a large model burn tokens on a trivial lookup, this feels like common sense finally catching up to engineering. The same logic appears in What Four Simple Models Reveal About Bike-Share Forecasting, where lightweight baselines matched or beat complex models on a 48-hour prediction task. Both cases make the same point: start simple, measure where simple works, and only escalate when the data demands it. The difference here is that Cloudflare's approach does not stop at classification, it routes. That turns a static accuracy benchmark into a dynamic system that learns which questions are easy and which are worth the compute cost. The result is a system that feels fast because it is fast where it matters most.

The connection to How to Turn an LLM Into a Faster Decision Engine is direct. That piece showed how decision models can make LLMs faster by treating them as components in a larger pipeline, not as standalone oracles. Cloudflare's confidence-score routing is a practical instantiation of that idea. Instead of asking one model to do everything, they let a cheap model handle the high-confidence cases and reserve the expensive model for the edge cases. The 0.9 threshold is not arbitrary, it is a tunable lever that balances speed against coverage. Lower it and you catch more questions but risk more errors. Raise it and you protect accuracy but send more traffic to the slow model. Getting that balance right is where the real engineering lives.

The specific takeaway is this: a confidence score is not just a debugging tool, it is a design parameter. If you are building an AI agent or a query system, you can set your own threshold based on your tolerance for error and your cost per token. That is a concrete lever, not a vague aspiration. The related work in Five Tests, 0.68 Seconds: How to Keep AI Agents Safe and Productive reinforces the same theme, safety and speed come from constraints, not from bigger models. The open question Cloudflare leaves us with is how low that confidence threshold can go before the routing overhead eats the savings. Watch for that number in production benchmarks. If it drops below 0.8 while maintaining 98% accuracy, the case for a single monolithic model gets much harder to defend.

From Towards Data Science

With a 0.9 confidence cutoff, Cloudflare's System One model handles the easy cases and routes the rest to a bigger, slower LLM like Luna.

The post Clairic Hit 98% Accuracy on 72.8% of Questions via Confidence Scores appeared first on Towards Data Science.

Read the original at Towards Data Science