mlsanityData to 13 Oct 2026

Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation

arXiv 2405.13622 ↗First posted 22 May 2024cs.CL cs.IRcited by 25

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

We propose a new method to measure the task-specific accuracy of Retrieval-Augmented Large Language Models (RAG). Evaluation is performed by scoring the RAG on an automatically-generated synthetic exam composed of multiple choice questions based on the corpus of documents associated with the task. Our method is an automated, cost-efficient, interpretable, and robust strategy to select the optimal components for a RAG system. We leverage Item Response Theory (IRT) to estimate the quality of an exam and its informativeness on task-specific accuracy. IRT also provides a natural way to iteratively improve the exam by eliminating the exam questions that are not sufficiently informative about a model's ability. We demonstrate our approach on four new open-ended Question-Answering tasks based on Arxiv abstracts, StackExchange questions, AWS DevOps troubleshooting guides, and SEC filings. In addition, our experiments reveal more general insights into factors impacting RAG performance like size, retrieval mechanism, prompting and fine-tuning. Most notably, our findings show that choosing the right retrieval algorithms often leads to bigger performance gains than simply using a larger language model.

Key facts. Cited by 25 papers in the map (data through 13 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 27 of its 76 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

27 of 76 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Benchmarking Large Language Models in Retrieval-Augmented Generation4 Sep

Other references · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,196
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,131
  7. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,435
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,286
  9. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,268
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 797
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 668
  12. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 413
14 more
  1. 2023GPTScore: Evaluate as You Desirecited by 361
  2. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 260
  3. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  4. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 202
  5. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  6. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  7. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 103
  8. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
  9. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 67
  10. 2019MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehensioncited by 60
  11. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 56
  12. 2019Deep-IRT: Make Deep Learning Based Knowledge Tracing Explainable Using Item Response Theorycited by 49
  13. 2022Efficient Nearest Neighbor Search for Cross-Encoder Models using Matrix Factorizationcited by 10
  14. 2022Interactive Visual Reasoning under Uncertaintycited by 8

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 7; 2025: 12; 2026: 6.

25 papers.

Sort:

By area:

  1. 2024TimeSeriesExam: A time series understanding examcited by 48
  2. 2025IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theorycited by 40
  3. 2024Evaluation of RAG Metrics for Question Answering in the Telecom Domaincited by 19
  4. 2025GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluationcited by 18
  5. 2025TimeSeriesGym: A Scalable Benchmark for (Time Series) Machine Learning Engineering Agentscited by 14
  6. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  7. 2025Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culturecited by 11
  8. 2025C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generationcited by 9
  9. 2025Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systemscited by 9
  10. 2024Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coveragecited by 8
  11. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  12. 2026TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scalecited by 6
13 more
  1. 2025Toward a unified framework for data-efficient evaluation of large language modelscited by 5
  2. 2026Doctor-RAG: A Failure-Aware Repair Framework for Agentic Retrieval-Augmented Generationcited by 5
  3. 2024COS-Mix: Cosine Similarity and Distance Fusion for Improved Information Retrievalcited by 4
  4. 2024Benchmarking LLMs for Environmental Review and Permittingcited by 4
  5. 2025Healthy LLMs? Benchmarking LLM Knowledge of UK Government Public Health Informationcited by 3
  6. 2025MUSS: Multilevel Subset Selection for Relevance and Diversitycited by 2
  7. 2025Data Swarms: Optimizable Generation of Synthetic Evaluation Datacited by 2
  8. 2026Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theorycited by 2
  9. 2024Enhancing E-commerce Product Title Translation with Retrieval-Augmented Generation and Large Language Modelscited by 0
  10. 2025TelcoAI: Advancing 3GPP Technical Specification Search through Agentic Multi-Modal Retrieval-Augmented Generationcited by 0
  11. 2026FD-RAG: Federated Dual-System Retrieval-Augmented Generationcited by 0
  12. 2026Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasonerscited by 0
  13. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Models
  2. 2025Evaluating the Retrieval Robustness of Large Language Models
  3. 2024Toward Optimal Search and Retrieval for RAG
  4. 2024Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems
  5. 2024Retrieval-Augmented Generation for Domain-Specific Question Answering: A Case Study on Pittsburgh and CMU
  6. 2024RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering
  7. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Level
  8. 2025On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems