When simple keyword search ties an 8-billion-parameter embedding model at picking agent tools, while running forty times faster on just a CPU, it's time to question how much complexity we actually need. A recent benchmark on the ToolRet dataset, testing up to 3,794 tools, found that BM25 matched the performance of BGE-large when given full tool documentation. This isn't a knock against embeddings; it's a practical reminder that sophisticated AI doesn't always outperform well-structured search. For anyone building agents that need to select the right function or API, this result matters.
We've seen similar dynamics play out in retrieval accuracy before. Our piece on When Search Context Swells, Retrieval Accuracy Drops 40 Points showed that adding more context can actively degrade results, even with advanced embedding models. The same principle surfaces here: feeding an agent well-organized, complete documentation for each tool can make a simple keyword matcher as effective as a heavy neural model. The takeaway is concrete: before reaching for the largest embedding model you can deploy, test whether a leaner search pipeline, one that treats tool descriptions as searchable documents, already solves your problem. The speed difference alone can transform a sluggish agent into a responsive one.
This isn't an argument against using AI models for tool selection. Embeddings excel at semantic matching when queries are vague or tools have overlapping purposes. But the benchmark reveals that for many real-world agent tasks, where tool names and descriptions are precise, keyword search is not a fallback, it's a competitive baseline. The practical implication for developers is clear: optimize your tool documentation first. Write clear, distinct descriptions. Include example usage patterns. Structure metadata so that a simple inverted index can find the right match. If you can achieve near-perfect retrieval with BM25, you free up compute budget for other parts of your agent pipeline, like validation or response generation.
We've also explored how local validation can offload work from larger models, as described in Validate AI Output Locally with Ollama, Gemma 4, and Nimble. The same philosophy applies here: lean components working in concert often outperform a single monolithic model. The question worth watching is how this finding scales. As tool inventories grow into the tens of thousands, will keyword search maintain its edge, or will the semantic flexibility of embeddings eventually pull ahead? For now, the answer is to measure before you assume. Run your own ToolRet-style test on your agent's tool set. You might find that the fastest solution is also the simplest one.