mlsanityData to 14 Oct 2026

Why Retrieval-Augmented Generation Fails: A Graph Perspective

arXiv 2605.14192 ↗First posted 13 May 2026cs.CL cs.AIcited by 0

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has become a powerful and widely used approach for improving large language models by grounding generation in retrieved evidence. However, RAG systems still produce incorrect answers in many cases. Why RAG fails despite having access to external information remains poorly understood. We present a model-internal study of retrieval-augmented generation that examines how retrieved evidence influences answer generation. Using circuit tracing, we construct attribution graphs that model the flow of information through transformer layers during decoding. These graphs represent interactions among retrieved context, intermediate model activations, and generated tokens, providing a graph, circuit-level view of how external evidence is integrated into the model's reasoning process across multiple question answering benchmarks, we observe consistent structural differences: correct predictions exhibit deeper reasoning paths, more distributed evidence flow, and a more structured pattern of local connectivity, while failed predictions show shallower, fragmented, and overly concentrated evidence flow. Building on these findings, we develop a graph-based error detection framework that uses attribution-graph topology features. Furthermore, we show that attribution graphs enable targeted interventions. By reinforcing question-constrained evidence grounding, we reshape internal routing so that answer generation remains guided by the question, leading to more effective integration of retrieved information and fewer errors.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 32 of its 45 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

32 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Survey16 Sep
  2. 2023Benchmarking Large Language Models in Retrieval-Augmented Generation4 Sep

Other references · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  5. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  7. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  8. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  9. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  10. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  11. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  12. 2024Graph Retrieval-Augmented Generation: A Surveycited by 324
18 more
  1. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 283
  2. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 257
  3. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  4. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  5. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  6. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 115
  7. 2024Reducing hallucination in structured outputs via Retrieval-Augmented Generationcited by 93
  8. 2025RAG vs. GraphRAG: A Systematic Evaluation and Key Insightscited by 53
  9. 2024Don't Forget to Connect! Improving RAG with Graph-based Rerankingcited by 49
  10. 2025Parametric Retrieval Augmented Generationcited by 38
  11. 2025Removal of Hallucination on Hallucination: Debate-Augmented RAGcited by 29
  12. 2024Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspectivecited by 20
  13. 2025Verifying Chain-of-Thought Reasoning via Its Computational Graphcited by 20
  14. 2025Empowering GraphRAG with Knowledge Filtering and Integrationcited by 15
  15. 2025Shifting from Ranking to Set Selection for Retrieval Augmented Generationcited by 15
  16. 2024W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answeringcited by 6
  17. 2025Reasoning by Exploration: A Unified Approach to Retrieval and Generation over Graphscited by 6
  18. 2025Uncovering Graph Reasoning in Decoder-only Transformers with Circuit Tracingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Graph-based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Survey
  2. 2026Relevance Is Not Sufficient Evidence: Detecting Evidence Gaps Before Generation in RAG
  3. 2026Assessing the Downstream Utility of Evidence-Aware Retrieval in RAG
  4. 2025Beyond Chunks and Graphs: Retrieval-Augmented Generation through Triplet-Driven Thinking
  5. 2026CUE-R: Beyond the Final Answer in Retrieval-Augmented Generation
  6. 2025Beyond Static Retrieval: Opportunities and Pitfalls of Iterative Retrieval in GraphRAG
  7. 2025GE-Chat: A Graph Enhanced RAG Framework for Evidential Response Generation of LLMs
  8. 2026When Iterative RAG Beats Ideal Evidence: A Diagnostic Study in Scientific Multi-hop Question Answering