mlsanityData to 7 Oct 2026

CRAG -- Comprehensive RAG Benchmark

arXiv 2406.04744 ↗First posted 7 Jun 2024cs.CLcited by 94

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has recently emerged as a promising solution to alleviate Large Language Model (LLM)'s deficiency in lack of knowledge. Existing RAG datasets, however, do not adequately represent the diverse and dynamic nature of real-world Question Answering (QA) tasks. To bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG), a factual question answering benchmark of 4,409 question-answer pairs and mock APIs to simulate web and Knowledge Graph (KG) search. CRAG is designed to encapsulate a diverse array of questions across five domains and eight question categories, reflecting varied entity popularity from popular to long-tail, and temporal dynamisms ranging from years to seconds. Our evaluation of this benchmark highlights the gap to fully trustworthy QA. Whereas most advanced LLMs achieve <=34% accuracy on CRAG, adding RAG in a straightforward manner improves the accuracy only to 44%. State-of-the-art industry RAG solutions only answer 63% of questions without any hallucination. CRAG also reveals much lower accuracy in answering questions regarding facts with higher dynamism, lower popularity, or higher complexity, suggesting future research directions. The CRAG benchmark laid the groundwork for a KDD Cup 2024 challenge and attracted thousands of participants and submissions. We commit to maintaining CRAG to serve research communities in advancing RAG solutions and general QA solutions. CRAG is available at https://github.com/facebookresearch/CRAG/.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 27 of its 44 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

27 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries27 Jan
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec

Other references · 25

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2023Mistral 7Bcited by 3,065
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  11. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  12. 2023Large Language Modelscited by 774
13 more
  1. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  2. 2023The Falcon Series of Open Language Modelscited by 526
  3. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  4. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  5. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  6. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  7. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  8. 2022Can large language models reason about medical questions?cited by 194
  9. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167
  10. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 130
  11. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  12. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  13. 2023VerifAI: Verified Generative AIcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 42; 2026: 38.

94 papers.

Sort:

By area:

  1. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  2. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  3. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
  4. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
  5. 2024Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generationcited by 78
  6. 2025Retrieval-Augmented Generation with Conflicting Evidencecited by 60
  7. 2024Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Modelscited by 57
  8. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systemscited by 50
  9. 2025SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledgecited by 44
  10. 2026DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agentscited by 44
  11. 2025The Amazon Nova Family of Models: Technical Report and Model Cardcited by 41
  12. 2025A Survey of Data Agents: Emerging Paradigm or Overstated Hype?cited by 30
38 more
  1. 2024MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistantscited by 24
  2. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  3. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 20
  4. 2024Evaluation of RAG Metrics for Question Answering in the Telecom Domaincited by 19
  5. 2025Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challengescited by 19
  6. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Servingcited by 18
  7. 2025Benchmarking Poisoning Attacks against Retrieval-Augmented Generationcited by 17
  8. 2025InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generationcited by 15
  9. 2024MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systemscited by 14
  10. 2025The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factualitycited by 14
  11. 2025Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usagecited by 12
  12. 2025Learning Facts at Scale with Active Readingcited by 11
  13. 2025AssoMem: Scalable Memory QA with Multi-Signal Associative Retrievalcited by 10
  14. 2025CRAG-MM: Multi-modal Multi-turn Comprehensive RAG Benchmarkcited by 9
  15. 2024A Hybrid RAG System with Comprehensive Enhancement on Complex Reasoningcited by 8
  16. 2024When Reasoning Meets Information Aggregation: A Case Study with Sports Narrativescited by 7
  17. 2024RAG-based Question Answering over Heterogeneous Data and Textcited by 7
  18. 2025Dialogue Benchmark Generation from Knowledge Graphs with Cost-Effective Retrieval-Augmented LLMscited by 7
  19. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  20. 2025RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented Generation Systemscited by 7
  21. 2025WearVox: An Egocentric Multichannel Voice Assistant Benchmark for Wearablescited by 7
  22. 2025HawkBench: Investigating Resilience of RAG Methods on Stratified Information-Seeking Taskscited by 6
  23. 2025Principled Context Engineering for RAG: Statistical Guarantees via Conformal Predictioncited by 6
  24. 2024CARROT: A Learned Cost-Constrained Retrieval Optimization System for RAGcited by 5
  25. 2025Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimizationcited by 5
  26. 2025syftr: Pareto-Optimal Generative AIcited by 5
  27. 2025HKRAG: Holistic Knowledge Retrieval-Augmented Generation Over Visually-Rich Documentscited by 5
  28. 2025AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learningcited by 5
  29. 2026CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoningcited by 5
  30. 2025BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactionscited by 4
  31. 2025Dynamic Context Adaptation for Consistent Role-Playing Agents with Retrieval-Augmented Generationscited by 4
  32. 2026FinRetrieval: A Benchmark for Financial Data Retrieval by AI Agentscited by 4
  33. 2024Level-Navi Agent: A Framework and benchmark for Chinese Web Search Agentscited by 3
  34. 2026Pix2Fact: When Vision Is Not Enough -- Benchmarking Fine-Grained VQA with Web Verification on High-Resolution Real-World Scenescited by 3
  35. 2026DynaRAG: Bridging Static and Dynamic Knowledge in Retrieval-Augmented Generationcited by 3
  36. 2026URAG: A Benchmark for Uncertainty Quantification in Retrieval-Augmented Large Language Modelscited by 3
  37. 2025Investigating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queriescited by 2
  38. 2025CTIConnect: A Benchmark for Retrieval-Augmented LLMs over Heterogeneous Cyber Threat Intelligencecited by 2

50 most cited of 94 All 94 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting the Solution of Meta KDD Cup 2024: CRAG
  2. 2024MARAGS: A Multi-Adapter System for Multi-Task Retrieval Augmented Generation Question Answering
  3. 2026Augmenting Question Answering with A Hybrid RAG Approach
  4. 2026SRAG: RAG with Structured Data Improves Vector Retrieval
  5. 2025T2-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation
  6. 2024Corrective Retrieval Augmented Generation
  7. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems
  8. 2025Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorgana