mlsanityData to 7 Oct 2026

ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems

arXiv 2311.09476 ↗First posted 16 Nov 2023cs.CL cs.AI cs.IRcited by 259

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Evaluating retrieval-augmented generation (RAG) systems traditionally relies on hand annotations for input queries, passages to retrieve, and responses to generate. We introduce ARES, an Automated RAG Evaluation System, for evaluating RAG systems along the dimensions of context relevance, answer faithfulness, and answer relevance. By creating its own synthetic training data, ARES finetunes lightweight LM judges to assess the quality of individual RAG components. To mitigate potential prediction errors, ARES utilizes a small set of human-annotated datapoints for prediction-powered inference (PPI). Across eight different knowledge-intensive tasks in KILT, SuperGLUE, and AIS, ARES accurately evaluates RAG systems while using only a few hundred human annotations during evaluation. Furthermore, ARES judges remain effective across domain shifts, proving accurate even after changing the type of queries and/or documents used in the evaluated RAG systems. We make our code and datasets publicly available on Github.

Key facts. Cited by 259 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 32 of its 47 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

32 of 47 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
20 more
  1. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  2. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  3. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  4. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  5. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  6. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  7. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  8. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  9. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  10. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  11. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  12. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  13. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  14. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  15. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  16. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  17. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  18. 2020Relevance-guided Supervision for OpenQA with ColBERTcited by 71
  19. 2023UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankerscited by 37
  20. 2023Retrieving Supporting Evidence for LLMs Generated Answerscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 69; 2025: 80; 2026: 109.

259 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  4. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  5. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 253
  6. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  7. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 115
  8. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencecited by 109
  9. 2024Evaluating Retrieval Quality in Retrieval-Augmented Generationcited by 90
  10. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  11. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  12. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
38 more
  1. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  2. 2024Lynx: An Open Source Hallucination Evaluation Modelcited by 55
  3. 2024AutoEval Done Right: Using Synthetic Data for Model Evaluationcited by 53
  4. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Modelscited by 50
  5. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 46
  6. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 44
  7. 2024Rationale-Guided Retrieval Augmented Generation for Medical Question Answeringcited by 41
  8. 2024RAGEval: Scenario Specific RAG Evaluation Dataset Generation Frameworkcited by 39
  9. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  10. 2024Long2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recallcited by 32
  11. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  12. 2025LettuceDetect: A Hallucination Detection Framework for RAG Applicationscited by 32
  13. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generationcited by 31
  14. 2024Prediction-Powered Ranking of Large Language Modelscited by 29
  15. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  16. 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agentscited by 29
  17. 2024OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domaincited by 28
  18. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontierscited by 26
  19. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25
  20. 2024Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generationcited by 25
  21. 2024Stratified Prediction-Powered Inference for Hybrid Language Model Evaluationcited by 24
  22. 2024OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generationcited by 24
  23. 2025SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reductioncited by 24
  24. 2025The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Modelscited by 24
  25. 2024On the Evaluation of Machine-Generated Reportscited by 23
  26. 2024HyPA-RAG: A Hybrid Parameter Adaptive Retrieval-Augmented Generation System for AI Legal and Policy Applicationscited by 22
  27. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 22
  28. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 22
  29. 2025Shrinking the Generation-Verification Gap with Weak Verifierscited by 21
  30. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
  31. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  32. 2024SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelinescited by 19
  33. 2024Evaluation of RAG Metrics for Question Answering in the Telecom Domaincited by 19
  34. 2024The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approachescited by 18
  35. 2025GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluationcited by 18
  36. 2025FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domaincited by 16
  37. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMscited by 16
  38. 2024Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generationcited by 15

50 most cited of 259 All 259 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation
  2. 2024Semantic Tokens in Retrieval Augmented Generation
  3. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  4. 2024GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering
  5. 2024RAGProbe: An Automated Approach for Evaluating RAG Applications
  6. 2025RAGSmith: A Framework for Finding the Optimal Composition of Retrieval-Augmented Generation Methods Across Datasets
  7. 2025mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs
  8. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Level