mlsanityData to 7 Oct 2026

When Knowledge Changes: Metamorphic Testing of RAG Systems with Mutations

arXiv 2607.26843 ↗First posted 29 Jul 2026cs.SEcited by 2

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG)-based LLM systems rely on external document corpora that can evolve and change over time. However, current evaluation methodologies (e.g., RAGAS) assess correctness against static snapshots, failing to detect faults when routine updates, factual changes, or noise alter the underlying data. We introduce a metamorphic testing framework that evaluates the consistency of RAG systems under corpus evolution. We formalise a fault taxonomy and 11 mutation operators that systematically perturb the system at both the pre-chunk (retrieval index) and post-chunk (retrieved context) levels. An empirical evaluation across five datasets and over 28k mutants reveals metamorphic violation rates of 4.9-10.2%. In a meta-evaluation against ground truth, our metamorphic oracle achieves F1 scores of 0.927-1.000, while the best RAGAS metric reaches only 0.570. Finally, we provide actionable insights into mitigating these faults through retrieval re-configuration, generator upgrades, and LLM-based reranking.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 15 of its 23 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

15 of 23 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey21 Apr
  2. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov

Other references · 13

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  5. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  6. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  7. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 153
  8. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencecited by 109
  9. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  10. 2024RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Contentcited by 26
  11. 2024Pandora's Box or Aladdin's Lamp: A Comprehensive Analysis Revealing the Role of RAG Noise in Large Language Modelscited by 12
  12. 2025MetaRAG: Metamorphic Testing for Hallucination Detection in RAG Systemscited by 2
1 more
  1. 2023The Inversive Relationship Between Bugs and Patches: An Empirical Studycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026What Would Fix This RAG Failure? Auditing Counterfactual Response with Paired Evidence Interventionscited by 0
  2. 2026When Correct Memory Goes Wrong: Fuzzing Persistent Memory Use in LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning
  2. 2026RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Models
  3. 2026RAGStress: A controlled benchmark for evaluating retrieval-augmented generation under knowledge-base degradation
  4. 2024Seven Failure Points When Engineering a Retrieval Augmented Generation System
  5. 2024RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
  6. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  7. 2024XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation
  8. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs