Retrieval benchmarks have a way of drifting from the job they're meant to do. You train a model to ace a public dataset, and somewhere along the way, the metric becomes the goal. The team at Qonto noticed this and decided to push back. Their new approach ties retrieval evaluation to something far more practical: whether the system actually surfaces the document that answers a product question. That's the kind of thinking that resonates with anyone who has watched a model look great on paper and then fail in production. It's also a useful reminder that our own exploration of real-world computer vision deployments follows a similar principle, where edge cases and practical constraints matter more than a leaderboard position.
The core insight here is refreshingly simple. Instead of relying on generic relevance scores that can be gamed or that correlate poorly with actual user intent, Qonto built a metric that measures proportionality between the query and the document's ability to answer it. That sounds like a subtle shift, but it changes what you optimize for. You're no longer chasing a model that finds "related" text; you're chasing one that finds the *right* text. This is the same kind of pragmatic thinking that drives the Forrester function's use in machine learning, where a mathematical tool is repurposed not for elegance but for a concrete, measurable outcome. Both efforts share a refusal to accept the default framing of a problem when that framing doesn't serve the end user.
What makes this work is the accompanying benchmarking dataset. You can't improve what you can't measure, and Qonto has built a dataset that reflects real product questions rather than synthetic queries. That's harder than it sounds. It requires curating examples where the "correct" answer is unambiguous, which forces you to confront the messy reality that some questions have multiple valid answers and some have none. The fact that they've open-sourced both the metric and the code is a welcome move. It lowers the barrier for others to test their own embedding models against a more honest baseline, and it invites scrutiny that generic benchmarks rarely receive.
Our take? This is a step toward making retrieval evaluation less performative and more predictive of real-world success. If you're building a search or Q&A system, don't just ask whether your embeddings rank well on a standard benchmark. Ask whether they would have found the document that actually solves the user's problem. That question is harder to answer, but it's the one that matters. The open question is whether this approach scales beyond Qonto's domain, and whether other teams will adopt it or retreat to the comfort of familiar metrics. For now, the takeaway is direct: if your retrieval pipeline is tuned to a benchmark, you're probably leaving value on the table. The next time you evaluate an embedding model, consider building a small, task-specific dataset that mirrors your real queries. It won't be as clean as a public leaderboard, but it will tell you what you actually need to know.