mlsanityData to 7 Oct 2026

Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation

arXiv 2409.12941 ↗First posted 19 Sep 2024cs.CLcited by 158

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have demonstrated significant performance improvements across various cognitive tasks. An emerging application is using LLMs to enhance retrieval-augmented generation (RAG) capabilities. These systems require LLMs to understand user queries, retrieve relevant information, and synthesize coherent and accurate responses. Given the increasing real-world deployment of such systems, comprehensive evaluation becomes crucial. To this end, we propose FRAMES (Factuality, Retrieval, And reasoning MEasurement Set), a high-quality evaluation dataset designed to test LLMs' ability to provide factual responses, assess retrieval capabilities, and evaluate the reasoning required to generate final answers. While previous work has provided datasets and benchmarks to evaluate these abilities in isolation, FRAMES offers a unified framework that provides a clearer picture of LLM performance in end-to-end RAG scenarios. Our dataset comprises challenging multi-hop questions that require the integration of information from multiple sources. We present baseline results demonstrating that even state-of-the-art LLMs struggle with this task, achieving 0.40 accuracy with no retrieval. The accuracy is significantly improved with our proposed multi-step retrieval pipeline, achieving an accuracy of 0.66 (>50% improvement). We hope our work will help bridge evaluation gaps and assist in developing more robust and capable RAG systems.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 30 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

26 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MoreHopQA: More Than Multi-hop Reasoning19 Jun
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec

Other references · 24

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
12 more
  1. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  2. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  3. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  4. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  5. 2019ELI5: Long Form Question Answeringcited by 567
  6. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  7. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
  8. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 253
  9. 2022RealTime QA: What's the Answer Right Now?cited by 239
  10. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  11. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  12. 2024Understanding the Effects of Iterative Prompting on Truthfulnesscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 67; 2026: 86.

158 papers.

Sort:

By area:

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2024DeepSeek-V3 Technical Reportcited by 724
  3. 2025Kimi K2: Open Agentic Intelligencecited by 523
  4. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  5. 2024Measuring short-form factuality in large language modelscited by 345
  6. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
  7. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  8. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  9. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  10. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 144
  11. 2025WebExplorer: Explore and Evolve for Training Long-Horizon Web Agentscited by 86
  12. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agentscited by 80
38 more
  1. 2025MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scalingcited by 72
  2. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 69
  3. 2025WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agentscited by 66
  4. 2025SealQA: Raising the Bar for Reasoning in Search-Augmented Language Modelscited by 65
  5. 2025Memory as Action: Autonomous Context Curation for Long-Horizon Agentic Taskscited by 62
  6. 2025SimpleDeepSearcher: Deep Information Seeking via Web-Powered Reasoning Trajectory Synthesiscited by 55
  7. 2025DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RLcited by 53
  8. 2025Tongyi DeepResearch Technical Reportcited by 53
  9. 2025Open Deep Search: Democratizing Search with Open-source Reasoning Agentscited by 46
  10. 2025QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learningcited by 46
  11. 2025SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledgecited by 44
  12. 2026DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agentscited by 44
  13. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 40
  14. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 40
  15. 2025Scaling Agents via Continual Pre-trainingcited by 39
  16. 2025Agentic Entropy-Balanced Policy Optimizationcited by 39
  17. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  18. 2025MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agentscited by 35
  19. 2025DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesiscited by 31
  20. 2025Characterizing Deep Research: A Benchmark and Formal Definitioncited by 29
  21. 2025SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agentscited by 29
  22. 2025MedBrowseComp: Benchmarking Medical Deep Research and Computer Usecited by 28
  23. 2026MiroThinker-1.7 & H1: Towards Heavy-Duty Research Agents via Verificationcited by 28
  24. 2026Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgetscited by 28
  25. 2025DeepDiver: Adaptive Search Intensity Scaling via Open-Web Reinforcement Learningcited by 23
  26. 2026SkillOrchestra: Learning to Route Agents via Skill Transfercited by 22
  27. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
  28. 2025YourBench: Easy Custom Evaluation Sets for Everyonecited by 20
  29. 2025MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documentscited by 20
  30. 2024Infogent: An Agent-Based Framework for Web Information Aggregationcited by 19
  31. 2025Sculptor: Empowering LLMs with Cognitive Agency via Active Context Managementcited by 17
  32. 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
  33. 2025DynaSearcher: Dynamic Knowledge Graph Augmented Search Agent via Multi-Reward Reinforcement Learningcited by 14
  34. 2025ReaRAG: Knowledge-guided Reasoning Enhances Factuality of Large Reasoning Models with Iterative Retrieval Augmented Generationcited by 13
  35. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 13
  36. 2026StackPlanner: A Centralized Hierarchical Multi-Agent System with Task-Experience Memory Managementcited by 13
  37. 2025DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generationcited by 12
  38. 2025ManuSearch: Democratizing Deep Search in Large Language Models with a Transparent and Open Multi-Agent Frameworkcited by 11

50 most cited of 158 All 158 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  2. 2026Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategies
  3. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Level
  4. 2024FS-RAG: A Frame Semantics Based Approach for Improved Factual Accuracy in Large Language Models
  5. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  6. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  7. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely
  8. 2025On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems