Explore smarter code retrieval with AI that understands

Explore smarter code retrieval with AI that understands structure, not just text.

In the realm of retrieval-augmented generation for large codebases, traditional methods often fall short due to their reliance on semantic similarity at the chunk level.

May 1, 20263 min readMachine Learning

In the rapidly evolving landscape of machine learning and code understanding, the approach to retrieval-augmented generation (RAG) using AST-derived graphs presents a significant advancement. Traditional methods often struggle with the complexities inherent in codebases, primarily due to the limitations of chunk-based text segmenting. The innovative use of Abstract Syntax Tree (AST)-derived graphs addresses a critical gap: capturing the structural relationships between code components. This is not merely a technical refinement; it fundamentally enhances how we think about data retrieval in programming contexts. For those wrestling with limitations in their own RAG systems, as discussed in articles like [Three limitations I keep hitting with retrieval-augmented generation in production and I'm running out of ideas [D]](/post/three-limitations-i-keep-hitting-with-retrieval-augmented-ge-cmoh5dq4c149vzxsxzrrakema), this method could serve as a beacon of potential solutions.

The AST-derived graph approach stands out because it aligns with the inherent nature of code: it is not just text but a structured representation of logic and relationships. By parsing code into a typed node and edge graph, the author effectively transforms static code into a dynamic, interrelated structure that mirrors the code's operational dependencies. This is crucial because, in programming, a function's utility often depends on its interactions with other functions and types across different files. The author emphasizes that by utilizing BM25 scoring over node metadata instead of relying solely on semantic embeddings, the method significantly reduces the context needed for large codebases—from approximately 100K tokens to about 5K tokens. This efficiency not only streamlines the retrieval process but also enhances the relevance of returned results, ultimately improving developer productivity.

Moreover, the methodology invites a deeper discussion about the appropriateness of retrieval techniques in different contexts. The decision to rely on BM25 rather than embedding similarity highlights the nuanced nature of code queries, where lexical distinctiveness plays a crucial role. This raises important questions about the future of retrieval systems: how can we continue to refine these techniques for various data types? There are still open questions about edge-weighting strategies and the potential benefits of cross-encoders. These considerations are vital for practitioners seeking to optimize their systems further.

Looking ahead, the implications of this approach extend beyond mere efficiency. They challenge developers and researchers to rethink their assumptions about data retrieval in programming contexts. What if we could leverage ASTs not only for code but also for other structured data formats? The potential for cross-disciplinary applications could redefine how we approach data management and retrieval across various fields. As we continue to innovate and explore these avenues, it will be fascinating to see how the community responds to these challenges and whether new paradigms will emerge in the realm of retrieval-augmented generation. For now, the conversation around AST-derived graphs and their applicability is one worth watching closely.

From Machine Learning

Wanted to share an approach I've been using for retrieval-augmented generation over large codebases and get feedback from people thinking about similar problems.
The problem Naive codebase RAG typically works by chunking files into text segments and embedding them for similarity search. This breaks down on code because semantic similarity at the chunk level doesn't capture structural relationships — a function in file A calling a type defined in file C won't surface that dependency through embedding proximity alone.

Beyond the 80% ceiling: Solving RAG's predictable blind spots in production

I've had a RAG system running in production for a few months now (legal domain, German regulatory documents). It handles 80% of queries well but there are three patterns where it fails predictably and I haven't found clean solutions. The scatter problem. Some questions need information from 8-10 different documents where each one contributes just a small piece. Vector search finds chunks related to the query but not chunks related to each other. So when someone asks something like "compare how notification deadlines work across different German federal states" the system finds 2-3 state-specific documents that happen to match the query well and misses the rest. The answer looks complete but it's actually partial. Cranking up k adds noise and burns tokens without reliably solving it because the missing documents might use completely different terminology for the same concept. I've thought about query decomposition (break the question into sub-queries per state) but that assumes the system knows upfront how many sub-queries to generate and what dimensions to decompose along. For a general-purpose research tool that feels brittle. The negative knowledge problem. When someone asks "do we have any guidance on employee monitoring" and the answer is genuinely no, the system can't cleanly say that. It retrieves whatever chunks are least irrelevant, and the LLM synthesizes something from them anyway. The user gets a confident-sounding answer about a tangentially related topic instead of a straightforward "this isn't covered in your knowledge base." I've tried similarity score thresholds as a gate but the problem is there's no clean boundary. A legitimate but unusual query might have low similarity scores. A genuinely off-topic query might match some chunks reasonably well because of shared vocabulary. Every threshold I've tested either filters out too much or too little. The prompt instruction to admit uncertainty helps maybe 60% of the time. The other 40% the model just reaches. The timeline problem. Questions like "how did the interpretation of X change after the 2023 ruling" require the system to find pre-ruling documents, find post-ruling documents, understand the temporal relationship, and construct a comparative narrative. The metadata has document dates. The prompt says to respect temporal ordering. But the model struggles to build a coherent before/after story when the retrieved chunks don't explicitly reference each other. It tends to either merge everything into one flat answer or just cite the newer source and ignore the older interpretation. This feels like it needs a fundamentally different retrieval approach (maybe temporal filtering at the search level, or separate retrievals for different time periods) rather than more prompt engineering. I've been reading about graph RAG approaches, agentic retrieval loops, and multi-hop reasoning chains but most of the literature is benchmarks on synthetic datasets, not production implementations. If anyone has actually deployed solutions for any of these three patterns I'd really like to hear what worked and what didn't. Especially interested in approaches that don't require restructuring the entire pipeline. submitted by /u/Fabulous-Pea-5366 [link] [comments]