mlsanityData to 14 Oct 2026

RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Models

arXiv 2608.00054 ↗First posted 26 Jul 2026cs.AI cs.SEcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to use external and domain-specific knowledge, but its reliability depends on the interaction between the generative model, embedding model, retrieval mechanism, and prompt construction strategy. We present RagTester, an automated end-to-end testing approach for RAG systems. RagTester generates retrieval documents, test inputs, and expected outputs; executes the tests; and evaluates the resulting answers using an LLM as a judge. Its test-generation strategy targets complex passages, unsupported queries, and document-coverage criteria. We evaluate RagTester using eight LLMs and six embedding models, yielding 24 compatible configurations, and compare it with a baseline test-input generator. Across 72,000 test executions, RagTester detected 21,633 failures, 6.6% more than the baseline, and outperformed it in 20 of the 24 configurations. The detected failures include inaccurate retrieval, unsupported answers, incomplete use of retrieved context, and difficulties interpreting complex passages. These results show that coverage-oriented test generation can effectively expose failures caused by the interaction between retrieval and generation components and support the assessment of RAG configurations before deployment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 34 of its 54 references that are in the map. First posted on arXiv 26 Jul 2026.

Builds on

34 of 54 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Testing Retrieval-Augmented Generation Systems with Chunk Coverage20 Jul
  2. 2025Towards Global Retrieval Augmented Generation: A Benchmark for Corpus-Level Reasoning30 Oct
  3. 2024RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework2 Aug
  4. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems25 Jun
  5. 2024CRAG -- Comprehensive RAG Benchmark7 Jun
  6. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models30 Jan
  7. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries27 Jan
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  9. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation26 Sep

Other references · 25

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  4. 2017Billion-scale similarity search with GPUscited by 2,497
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  8. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  9. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  10. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  11. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  12. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
13 more
  1. 2024The Faiss librarycited by 376
  2. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 341
  3. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  4. 2022Teaching language models to support answers with verified quotescited by 292
  5. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  6. 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domaincited by 118
  7. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systemscited by 51
  8. 2021Recall@k Surrogate Loss with Large Batches and Similarity Mixupcited by 49
  9. 2025Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievalscited by 28
  10. 2025RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented Generation Systemscited by 8
  11. 2025Redefining Retrieval Evaluation in the Era of LLMscited by 3
  12. 2025Meta-Fair: AI-Assisted Fairness Testing of Large Language Modelscited by 1
  13. 2026Retromorphic Testing with Hierarchical Verification for Hallucination Detection in RAGcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning
  2. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  3. 2024RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation
  4. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
  5. 2024Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation
  6. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Models
  7. 2024Toward Optimal Search and Retrieval for RAG
  8. 2024XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation