The "lost in the middle" effect has become one of those cited-but-rarely-tested claims in AI, a paper finding that models lose track of information placed in the middle of a long context. A developer recently decided to check it on actual models rather than repeat the citation, and the results are worth paying attention to. Using made-up facts that models couldn't have memorized, they slid a single answer-bearing document through positions among 5, 15, and 30 distractor documents, asking the same question each time. On Claude Haiku 4.5 and Sonnet 5, they found zero position-based flips at 5 and 15 documents. At 30 documents, an apparent 4% flip rate turned out to be the model answering "3" versus "three" for the same correct answer, a formatting difference, not a retrieval failure. Once they compared meaning instead of strings, the rate dropped back to zero.
This is the kind of practical check we need more of, especially when a finding becomes dogma. The developer is honest about the caveats: this was easy single-fact retrieval with short documents. The effect likely still appears with long documents, multi-hop reasoning, or ambiguous relevance. But the fact that two current models showed no position sensitivity on straightforward retrieval up to 30 documents is genuinely useful information. It suggests that for a large class of real-world use cases, pulling a specific fact from a moderate pile of short references, the "lost in the middle" problem may be largely solved, at least on these models. This aligns with what we've seen in other recent tests, like When a Tiny GPT Knows the Facts but Misses the Connection, where models can memorize facts but fail when asked to combine them. The failure mode is shifting from position sensitivity to something more subtle: not where the fact sits, but how it connects to others.
What this means for anyone building on these models is straightforward: if your task is simple retrieval from a moderate context, you can probably stop worrying about document order as a primary failure mode. But the developer's experience with the "3" versus "three" trap is a concrete warning. Exact string matching will lie to you. If you're evaluating your own system, compare meaning, not strings. The script they shared on GitHub makes it easy to run this test yourself, and we'd encourage anyone relying on retrieval to do exactly that. The real open question is where the effect does reappear, at what context length, with what document complexity, and on which models. That's the test worth running next, and it's the one that will tell us where the actual boundaries are.