mlsanityData to 7 Oct 2026

Evaluating Quality of Answers for Retrieval-Augmented Generation: A Strong LLM Is All You Need

arXiv 2406.18064 ↗First posted 26 Jun 2024cs.CLcited by 7

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

We present a comprehensive study of answer quality evaluation in Retrieval-Augmented Generation (RAG) applications using vRAG-Eval, a novel grading system that is designed to assess correctness, completeness, and honesty. We further map the grading of quality aspects aforementioned into a binary score, indicating an accept or reject decision, mirroring the intuitive "thumbs-up" or "thumbs-down" gesture commonly used in chat applications. This approach suits factual business contexts where a clear decision opinion is essential. Our assessment applies vRAG-Eval to two Large Language Models (LLMs), evaluating the quality of answers generated by a vanilla RAG application. We compare these evaluations with human expert judgments and find a substantial alignment between GPT-4's assessments and those of human experts, reaching 83% agreement on accept or reject decisions. This study highlights the potential of LLMs as reliable evaluators in closed-domain, closed-ended settings, particularly when human evaluations require significant resources.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 11 of its 17 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

11 of 17 references.

References · 11

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  7. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  8. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  9. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
  10. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53
  11. 2021Semantic Answer Similarity for Evaluating Question Answering Modelscited by 47

Built on by

Cited by year

Citing papers per year: 2025: 7.

7 papers.

  1. 2025MultiFinRAG: An Optimized Multimodal Retrieval-Augmented Generation (RAG) Framework for Financial Question Answeringcited by 9
  2. 2025Generative AI in Transportation Planning: A Surveycited by 8
  3. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  4. 2025RAGXplain: From Explainable Evaluation to Actionable Guidance of RAG Pipelinescited by 4
  5. 2025HGMEM: Hypergraph-based Working Memory to Improve Multi-step RAG for Long-Context Complex Relational Modelingcited by 4
  6. 2025How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAGcited by 2
  7. 2025GRASP: Municipal Budget AI Chatbots for Enhancing Civic Engagementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering
  2. 2025Chatbot Arena Meets Nuggets: Towards Explanations and Diagnostics in the Evaluation of LLM Responses
  3. 2025Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality
  4. 2025A Knowledge Graph and a Tripartite Evaluation Framework Make Retrieval-Augmented Generation Scalable and Transparent
  5. 2025MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generation
  6. 2025Evaluation of retrieval-based QA on QUEST-LOFT
  7. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
  8. 2026The RAT: A Unified Bayesian Model for RAG Evaluation