mlsanityData to 14 Oct 2026

RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation

arXiv 2508.06056 ↗First posted 8 Aug 2025cs.HCcited by 4

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems have emerged as a promising solution to enhance large language models (LLMs) by integrating external knowledge retrieval with generative capabilities. While significant advancements have been made in improving retrieval accuracy and response quality, a critical challenge remains that the internal knowledge integration and retrieval-generation interactions in RAG workflows are largely opaque. This paper introduces RAGTrace, an interactive evaluation system designed to analyze retrieval and generation dynamics in RAG-based workflows. Informed by a comprehensive literature review and expert interviews, the system supports a multi-level analysis approach, ranging from high-level performance evaluation to fine-grained examination of retrieval relevance, generation fidelity, and cross-component interactions. Unlike conventional evaluation practices that focus on isolated retrieval or generation quality assessments, RAGTrace enables an integrated exploration of retrieval-generation relationships, allowing users to trace knowledge sources and identify potential failure cases. The system's workflow allows users to build, evaluate, and iterate on retrieval processes tailored to their specific domains of interest. The effectiveness of the system is demonstrated through case studies and expert evaluations on real-world RAG applications.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 45 of its 73 references that are in the map. First posted on arXiv 8 Aug 2025.

Builds on

45 of 73 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective1 Oct
  2. 2024RAGProbe: An Automated Approach for Evaluating RAG Applications24 Sep
  3. 2024Evaluation of Retrieval-Augmented Generation: A Survey13 May
  4. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models10 May
  5. 2024Evaluating Retrieval Quality in Retrieval-Augmented Generation21 Apr
  6. 2024Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrievers22 Mar
  7. 2024Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge3 Mar
  8. 2024The Power of Noise: Redefining Retrieval for RAG Systems26 Jan
  9. 2024Seven Failure Points When Engineering a Retrieval Augmented Generation System11 Jan
  10. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov
  11. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation26 Sep

Other references · 34

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  2. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 517
  3. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  4. 2023A Survey on Large Language Models for Recommendationcited by 379
  5. 2023Explainability for Large Language Models: A Surveycited by 333
  6. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  7. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 149
  8. 2022Dense Text Retrieval based on Pretrained Language Models: A Surveycited by 136
  9. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 111
  10. 2024HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extractioncited by 93
  11. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  12. 2023ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testingcited by 88
22 more
  1. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 74
  2. 2023Automatic Evaluation of Attribution by Large Language Modelscited by 72
  3. 2024CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Modelscited by 54
  4. 2022Retrieval-Enhanced Machine Learningcited by 47
  5. 2024WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualizationcited by 39
  6. 2024Knowledge Graphs as Context Sources for LLM-Based Explanations of Learning Recommendationscited by 31
  7. 2024Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuningcited by 31
  8. 2024LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Modelscited by 29
  9. 2023The HaLLMark Effect: Supporting Provenance and Transparent Use of Large Language Models in Writing with Interactive Visualizationcited by 24
  10. 2023RELIC: Investigating Large Language Model Responses using Self-Consistencycited by 23
  11. 2024KNOWNET: Guided Health Information Seeking from LLMs via Knowledge Graph Integrationcited by 15
  12. 2023When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTscited by 13
  13. 2023Angler: Helping Machine Translation Practitioners Prioritize Model Improvementscited by 12
  14. 2024KnowledgeVIS: Interpreting Language Models by Comparing Fill-in-the-Blank Promptscited by 7
  15. 2024iScore: Visual Analytics for Interpreting How Language Models Automatically Score Summariescited by 6
  16. 2024LinkQ: An LLM-Assisted Visual Interface for Knowledge Graph Question-Answeringcited by 6
  17. 2023DeepLens: Interactive Out-of-distribution Data Detection in NLP Modelscited by 5
  18. 2024Knowledge graph enhanced retrieval-augmented generation for failure mode and effects analysiscited by 5
  19. 2023From Information to Choice: A Critical Inquiry Into Visualization Tools for Decision Makingcited by 4
  20. 2024Towards Dataset-scale and Feature-oriented Evaluation of Text Summarization in Large Language Model Promptscited by 3
  21. 2024"The Data Says Otherwise"-Towards Automated Fact-checking and Communication of Data Claimscited by 3
  22. 2024Enhancing Retrieval Processes for Language Generation with Augmented Queriescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026A Systematic Investigation of Document Chunking Strategies and Embedding Sensitivitycited by 3
  2. 2026RAGExplorer: A Visual Analytics System for the Comparative Diagnosis of RAG Systemscited by 2
  3. 2026Neurosymbolic Retrievers for Retrieval-augmented Generationcited by 1
  4. 2026Hierarchical Long-Term Semantic Memory for LinkedIn's Hiring Agentcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generation
  2. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  3. 2025Dynamic and Parametric Retrieval-Augmented Generation
  4. 2025Enhancing Retrieval-Augmented Generation: A Study of Best Practices
  5. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  6. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  7. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning
  8. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs