mlsanityData to 14 Oct 2026

MIRAGE: A Metric-Intensive Benchmark for Retrieval-Augmented Generation Evaluation

arXiv 2504.17137 ↗First posted 23 Apr 2025cs.CL cs.AIcited by 11

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has gained prominence as an effective method for enhancing the generative capabilities of Large Language Models (LLMs) through the incorporation of external knowledge. However, the evaluation of RAG systems remains a challenge, due to the intricate interplay between retrieval and generation components. This limitation has resulted in a scarcity of benchmarks that facilitate a detailed, component-specific assessment. In this work, we present MIRAGE, a Question Answering dataset specifically designed for RAG evaluation. MIRAGE consists of 7,560 curated instances mapped to a retrieval pool of 37,800 entries, enabling an efficient and precise evaluation of both retrieval and generation tasks. We also introduce novel evaluation metrics aimed at measuring RAG adaptability, encompassing dimensions such as noise vulnerability, context acceptability, context insensitivity, and context misinterpretation. Through comprehensive experiments across various retriever-LLM configurations, we provide new insights into the optimal alignment of model pairs and the nuanced dynamics within RAG systems. The dataset and evaluation code are publicly available, allowing for seamless integration and customization in diverse research settingsfootnoteThe MIRAGE code and data are available at https://github.com/nlpai-lab/MIRAGE.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 29 of its 41 references that are in the map. First posted on arXiv 23 Apr 2025.

Builds on

29 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Survey29 Feb
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  3. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov
  4. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation28 Sep

Other references · 25

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2024Qwen2 Technical Reportcited by 2,120
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  10. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  11. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  12. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
13 more
  1. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  2. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  3. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  4. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  5. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  6. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  7. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  8. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 298
  9. 2022RealTime QA: What's the Answer Right Now?cited by 244
  10. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  11. 2023RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledgecited by 51
  12. 2023IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositionscited by 26
  13. 2020HopRetriever: Retrieve Hops over Wikipedia to Answer Complex Questionscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 9.

11 papers.

  1. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  2. 2026Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questionscited by 3
  3. 2026MiRAGE: A Multiagent Framework for Generating Multimodal Multihop Question-Answer Dataset for RAG Evaluationcited by 1
  4. 2026Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarkscited by 1
  5. 2026Twin Worlds: Equivariance-Based Abstention for Evidence-Grounded Reasoningcited by 1
  6. 2025LFD: Layer Fused Decoding to Exploit External Knowledge in Retrieval-Augmented Generationcited by 0
  7. 2026Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)cited by 0
  8. 2026Self-evolving Agentic Customer Support System at LinkedIncited by 0
  9. 2026RAG-Safety-Bench: Reliable Evaluation of Retrieval-Augmented LLM Safetycited by 0
  10. 2026Beyond Benchmark Scores: How Synthetic and Authentic Query Distributions Diverge in RAG Evaluationcited by 0
  11. 2026The Copy Ceiling: An Input-Exposure Control for Ontology-Grounded Generation over Curated Corporacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  2. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  3. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  4. 2024Evaluation of Retrieval-Augmented Generation: A Survey
  5. 2024MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems
  6. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  7. 2024CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity
  8. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Level