Three limitations I keep hitting with retrieval-augmented generation in production and I'm running out of ideas [D]
Our take
The challenges of implementing retrieval-augmented generation (RAG) systems in production, particularly in specialized fields like legal documentation, highlight both the potential and the limitations of current AI technologies. As illustrated in the recent article “Three limitations I keep hitting with retrieval-augmented generation in production,” the author grapples with three significant issues: the scatter problem, the negative knowledge problem, and the timeline problem. These hurdles not only impede efficiency but also raise important questions about the reliability of AI in handling complex queries, emphasizing the need for continuous innovation in this space.
The scatter problem, where information is dispersed across multiple documents, illustrates a significant limitation in the current RAG frameworks. When users pose questions that require a nuanced understanding of multiple sources, the system often fails to provide a comprehensive answer, resulting in incomplete information. This is particularly problematic in legal contexts where precision is crucial. The potential solution of query decomposition may seem promising, but as the author notes, it relies on the system's ability to anticipate the necessary dimensions of inquiry—a task that can be both brittle and impractical. This situation calls for a more robust approach to data retrieval, perhaps through methods like graph-based RAG systems or agentic retrieval loops, as explored in related discussions such as “RAG Is Blind to Time — I Built a Temporal Layer to Fix It in Production.”
The negative knowledge problem further complicates user interactions with AI systems. When a query yields no relevant information, users expect a clear, honest response. Instead, they often receive vague or tangentially related answers that undermine trust in the system’s capabilities. This issue highlights the importance of transparency in AI responses. Implementing similarity score thresholds as a filter can help, but the nuances of language and context often defy straightforward categorization. The challenge here is not just technical; it is also a question of how we can design systems that prioritize user clarity and understanding over mere data retrieval.
Moreover, the timeline problem underscores the limitations of current retrieval methods in understanding temporal relationships between documents. Users asking for comparative analyses across time periods require a system that not only retrieves relevant documents but also synthesizes a coherent narrative from them. The current models' struggles to construct these narratives signal a gap in our understanding of how to effectively integrate temporal data within RAG frameworks. As the author suggests, this might necessitate a reevaluation of retrieval strategies, including the potential for temporal filters that could enhance the system's contextual understanding.
As we look ahead, the challenges articulated in this discourse serve as a crucial reminder of the ongoing journey in AI development. The quest for more intuitive and effective data management solutions is not just a technical endeavor but a human-centered one, focused on improving user outcomes. It is essential for the AI community to engage with these limitations candidly, exploring innovative approaches that not only address current shortcomings but also pave the way for future advancements. As we continue to push the boundaries of what AI can achieve, the question remains: how can we build systems that are not only powerful but also trustworthy in their responses? This inquiry will likely define the next phase of AI evolution in data management.
I've had a RAG system running in production for a few months now (legal domain, German regulatory documents). It handles 80% of queries well but there are three patterns where it fails predictably and I haven't found clean solutions.
The scatter problem.
Some questions need information from 8-10 different documents where each one contributes just a small piece. Vector search finds chunks related to the query but not chunks related to each other. So when someone asks something like "compare how notification deadlines work across different German federal states" the system finds 2-3 state-specific documents that happen to match the query well and misses the rest. The answer looks complete but it's actually partial. Cranking up k adds noise and burns tokens without reliably solving it because the missing documents might use completely different terminology for the same concept.
I've thought about query decomposition (break the question into sub-queries per state) but that assumes the system knows upfront how many sub-queries to generate and what dimensions to decompose along. For a general-purpose research tool that feels brittle.
The negative knowledge problem.
When someone asks "do we have any guidance on employee monitoring" and the answer is genuinely no, the system can't cleanly say that. It retrieves whatever chunks are least irrelevant, and the LLM synthesizes something from them anyway. The user gets a confident-sounding answer about a tangentially related topic instead of a straightforward "this isn't covered in your knowledge base."
I've tried similarity score thresholds as a gate but the problem is there's no clean boundary. A legitimate but unusual query might have low similarity scores. A genuinely off-topic query might match some chunks reasonably well because of shared vocabulary. Every threshold I've tested either filters out too much or too little. The prompt instruction to admit uncertainty helps maybe 60% of the time. The other 40% the model just reaches.
The timeline problem.
Questions like "how did the interpretation of X change after the 2023 ruling" require the system to find pre-ruling documents, find post-ruling documents, understand the temporal relationship, and construct a comparative narrative. The metadata has document dates. The prompt says to respect temporal ordering. But the model struggles to build a coherent before/after story when the retrieved chunks don't explicitly reference each other. It tends to either merge everything into one flat answer or just cite the newer source and ignore the older interpretation.
This feels like it needs a fundamentally different retrieval approach (maybe temporal filtering at the search level, or separate retrievals for different time periods) rather than more prompt engineering.
I've been reading about graph RAG approaches, agentic retrieval loops, and multi-hop reasoning chains but most of the literature is benchmarks on synthetic datasets, not production implementations. If anyone has actually deployed solutions for any of these three patterns I'd really like to hear what worked and what didn't. Especially interested in approaches that don't require restructuring the entire pipeline.
[link] [comments]
Read on the original site
Open the publisher's page for the full experience
Related Articles
- RAG Is Blind to Time — I Built a Temporal Layer to Fix It in ProductionThree weeks into testing, a learner told me my AI tutor gave her the wrong answer. Not obviously wrong — just outdated enough to mislead. That was the moment I realized something most RAG systems quietly ignore: they have no sense of time. My system retrieved the most similar document, not the most current one. And in a knowledge base that changes constantly, that’s a serious flaw. The fix wasn’t in the retriever or the model. It was in the gap between them. I built a temporal layer that filters expired facts, boosts time-sensitive signals, and makes the system prefer what’s still true — not just what matches. The post RAG Is Blind to Time — I Built a Temporal Layer to Fix It in Production appeared first on Towards Data Science.
- Codebase-scale retrieval using AST-derived graphs + BM25 — reducing LLM context from 100K to 5K tokens [D]Wanted to share an approach I've been using for retrieval-augmented generation over large codebases and get feedback from people thinking about similar problems. The problem Naive codebase RAG typically works by chunking files into text segments and embedding them for similarity search. This breaks down on code because semantic similarity at the chunk level doesn't capture structural relationships — a function in file A calling a type defined in file C won't surface that dependency through embedding proximity alone. The approach: AST-derived typed graphs Instead of chunking, I parse every file using Tree-sitter into its AST, then extract a typed node/edge graph: Nodes: functions, classes, interfaces, types, modules Edges: imports, exports, call relationships, inheritance, composition This gets stored in SQLite as a persistent graph. Parse cost is one-time per project. Retrieval: BM25 over graph nodes At query time, instead of embedding similarity, I run BM25 scoring over node metadata (names, signatures, docstrings, file paths). Top-scoring nodes get passed to the LLM. The graph structure means a retrieved function automatically pulls in its direct dependencies via edge traversal. Empirically this lands at ~5K tokens per query on medium-large codebases that would otherwise require ~100K tokens with naive full-context approaches. Hierarchical fallback for complex queries For multi-file reasoning tasks: A Mermaid diagram of the full graph serves as a persistent architectural map always in context BM25 node retrieval handles targeted lookup At 70% context capacity, a fast model compresses least-relevant nodes before passing to the primary model Why BM25 over embeddings here Code identifiers (function names, type names, module paths) are highly distinctive lexically. BM25 outperforms embedding similarity on exact and near-exact identifier matching, which is the dominant retrieval pattern in code queries. Embeddings would likely help more for natural language docstring queries — haven't benchmarked that comparison rigorously yet. Open questions I'm still thinking about: Better edge-weighting strategies for the graph — currently all edges are unweighted Whether re-ranking with a cross-encoder would meaningfully improve precision over BM25 alone Handling dynamic languages where call graphs can't be fully resolved statically Has anyone tackled codebase-scale RAG differently? Particularly curious if anyone's compared AST-graph approaches against embedding-based chunk retrieval on real codebases with quantitative benchmarks. submitted by /u/Altruistic_Night_327 [link] [comments]