mlsanityData to 7 Oct 2026

VERA: Validation and Evaluation of Retrieval-Augmented Systems

arXiv 2409.03759 ↗First posted 16 Aug 2024cs.IR cs.AIcited by 4

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

The increasing use of Retrieval-Augmented Generation (RAG) systems in various applications necessitates stringent protocols to ensure RAG systems accuracy, safety, and alignment with user intentions. In this paper, we introduce VERA (Validation and Evaluation of Retrieval-Augmented Systems), a framework designed to enhance the transparency and reliability of outputs from large language models (LLMs) that utilize retrieved information. VERA improves the way we evaluate RAG systems in two important ways: (1) it introduces a cross-encoder based mechanism that encompasses a set of multidimensional metrics into a single comprehensive ranking score, addressing the challenge of prioritizing individual metrics, and (2) it employs Bootstrap statistics on LLM-based metrics across the document repository to establish confidence bounds, ensuring the repositorys topical coverage and improving the overall reliability of retrieval systems. Through several use cases, we demonstrate how VERA can strengthen decision-making processes and trust in AI applications. Our findings not only contribute to the theoretical understanding of LLM-based RAG evaluation metric but also promote the practical implementation of responsible AI systems, marking a significant advancement in the development of reliable and transparent generative AI technologies.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 17 of its 48 references that are in the map. First posted on arXiv 16 Aug 2024.

Builds on

17 of 48 references.

References · 17

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  8. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  9. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  10. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  11. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  12. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
5 more
  1. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  2. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53
  3. 2023RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledgecited by 51
  4. 2024Benchmarking Llama2, Mistral, Gemma and GPT for Factuality, Toxicity, Bias and Propensity for Hallucinationscited by 8
  5. 2023Enhancing Documents with Multidimensional Relevance Statements in Cross-encoder Re-rankingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  2. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  3. 2026Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategiescited by 1
  4. 2026Automating Multi-Hop RAG Evaluation via TRIAD: From Context Extraction to Validated Dataset Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024VERA: Validation and Enhancement for Retrieval Augmented systems
  2. 2024The Power of Noise: Redefining Retrieval for RAG Systems
  3. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
  4. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation
  5. 2024Semantic Tokens in Retrieval Augmented Generation
  6. 2024Evaluation of Retrieval-Augmented Generation: A Survey
  7. 2024Deploying Large Language Models With Retrieval Augmented Generation
  8. 2024Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDFs: An Experience Report