mlsanityData to 7 Oct 2026

Ragas: Automated Evaluation of Retrieval Augmented Generation

arXiv 2309.15217 ↗First posted 26 Sep 2023cs.CLcited by 408

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

We introduce Ragas (Retrieval Augmented Generation Assessment), a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines. RAG systems are composed of a retrieval and an LLM based generation module, and provide LLMs with knowledge from a reference textual database, which enables them to act as a natural language layer between a user and textual databases, reducing the risk of hallucinations. Evaluating RAG architectures is, however, challenging because there are several dimensions to consider: the ability of the retrieval system to identify relevant and focused context passages, the ability of the LLM to exploit such passages in a faithful way, or the quality of the generation itself. With Ragas, we put forward a suite of metrics which can be used to evaluate these different dimensions textitwithout having to rely on ground truth human annotations. We posit that such a framework can crucially contribute to faster evaluation cycles of RAG architectures, which is especially important given the fast adoption of LLMs.

Key facts. Cited by 408 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 27 of its 30 references that are in the map. First posted on arXiv 26 Sep 2023.

Builds on

27 of 30 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  10. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  11. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,050
15 more
  1. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  2. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  3. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  4. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  5. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  6. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  7. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  8. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  9. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  10. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  11. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  12. 2023GPTScore: Evaluate as You Desirecited by 357
  13. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  14. 2023Interpretable Unified Language Checkingcited by 26
  15. 2023KNN-LM Does Not Improve Open-ended Text Generationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 76; 2025: 125; 2026: 205.

408 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  3. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  4. 2024LightRAG: Simple and Fast Retrieval-Augmented Generationcited by 310
  5. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 253
  6. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  7. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  8. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 115
  9. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencecited by 109
  10. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 100
  11. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
  12. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
38 more
  1. 2025HyperGraphRAG: Retrieval-Augmented Generation via Hypergraph-Structured Knowledge Representationcited by 68
  2. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  3. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generationcited by 59
  4. 2024Lynx: An Open Source Hallucination Evaluation Modelcited by 55
  5. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systemscited by 50
  6. 2025Retrieval-Augmented Generation with Hierarchical Knowledgecited by 46
  7. 2024RAG-Fusion: a New Take on Retrieval-Augmented Generationcited by 42
  8. 2023PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuningcited by 39
  9. 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agentscited by 29
  10. 2024Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generationcited by 25
  11. 2025IntellAgent: A Multi-Agent Framework for Evaluating Conversational AI Systemscited by 25
  12. 2024OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generationcited by 24
  13. 2025SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reductioncited by 24
  14. 2025The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Modelscited by 24
  15. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  16. 2024CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systemscited by 23
  17. 2025Hyper-RAG: Combating LLM Hallucinations using Hypergraph-Driven Retrieval-Augmented Generationcited by 23
  18. 2024LOCALINTEL: Generating Organizational Threat Intelligence from Global and Local Cyber Knowledgecited by 22
  19. 2025Shrinking the Generation-Verification Gap with Weak Verifierscited by 21
  20. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  21. 2024Evaluation of RAG Metrics for Question Answering in the Telecom Domaincited by 19
  22. 2024Multi-Head RAG: Solving Multi-Aspect Problems with LLMscited by 18
  23. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  24. 2025FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generationcited by 17
  25. 2025MarketSenseAI 2.0: Enhancing Stock Analysis through LLM Agentscited by 16
  26. 2025RAS: Retrieval-And-Structuring for Knowledge-Intensive LLM Generationcited by 16
  27. 2024FACTS About Building Retrieval Augmented Generation-based Chatbotscited by 15
  28. 2024Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generationcited by 15
  29. 2025CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answeringcited by 15
  30. 2026Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluationcited by 15
  31. 2025Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignmentcited by 14
  32. 2025Auto-ARGUE: LLM-Based Report Generation Evaluationcited by 14
  33. 2024RetrievalQA: Assessing Adaptive Retrieval-Augmented Generation for Short-form Open-Domain Question Answeringcited by 13
  34. 2026From Spark to Fire: Modeling and Mitigating Error Cascades in LLM-Based Multi-Agent Collaborationcited by 13
  35. 2024Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systemscited by 12
  36. 2024From Questions to Insightful Answers: Building an Informed Chatbot for University Resourcescited by 11
  37. 2025Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systemscited by 11
  38. 2024Towards Optimizing and Evaluating a Retrieval Augmented QA Chatbot using LLMs with Human in the Loopcited by 10

50 most cited of 408 All 408 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  2. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  3. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  4. 2024CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity
  5. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey
  6. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
  7. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models
  8. 2024RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation