mlsanityData to 7 Oct 2026

Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA

arXiv 2608.22856 ↗First posted 24 Aug 2026cs.IR cs.CLcited by 1

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

A retrieval-augmented QA system can return different answers after an index expansion even when its requested model identifier, prompt, retrieval policy, evidence depth, rendering, and exposed generation controls are held fixed. Aggregate accuracy may hide these changes when gains and losses cancel, while ordinary generation variability makes one-shot comparisons overstate update effects. We call the hidden phenomenon accuracy-blind answer churn and introduce the Snapshot Compatibility Audit, which estimates excess answer churn by subtracting same-snapshot repeat disagreement from cross-snapshot disagreement. We instantiate it by expanding one frozen FineWeb prefix from one to seven shards. In a preregistered 400-question Natural Questions study, normalized-exact and blinded-semantic excess churn are 6.44 and 10.25 percentage points while exact-match accuracy changes by only -1.50 points. A post-hoc analysis finds repeat-stable semantic flips on 40/400 questions. A separately preregistered 200-question TriviaQA study yields smaller, directionally consistent excess churn while exact-match accuracy moves in the opposite direction. An outcome-blind post-hoc 100-question subset replication with a second DeepSeek generator and serving configuration finds 8.75 pp of semantic excess churn even as exact match rises by 3.00 percentage points. Answer-level compatibility can therefore fail without a conspicuous or consistently directed utility shift. Retrieval-augmented releases should audit compatibility alongside utility.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 45 of its 78 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

45 of 78 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection17 Oct
  2. 2023Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise2 May

Other references · 43

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  11. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,050
31 more
  1. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  2. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  3. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  4. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  5. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  6. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  7. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  8. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  9. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  10. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  11. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  12. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  13. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  14. 2023How is ChatGPT's behavior changing over time?cited by 265
  15. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  16. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 180
  17. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  18. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 123
  19. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 93
  20. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Studycited by 63
  21. 2024StreamBench: Towards Benchmarking Continuous Improvement of Language Agentscited by 44
  22. 2023Large Language Models Are Semi-Parametric Reinforcement Learning Agentscited by 42
  23. 2024MUSCLE: A Model Update Strategy for Compatible LLM Evolutioncited by 15
  24. 2021Backward-Compatible Prediction Updates: A Probabilistic Approachcited by 14
  25. 2022Measuring and Reducing Model Update Regression in Structured Prediction for NLPcited by 11
  26. 2024Toward Robust RALMs: Revealing the Impact of Imperfect Retrieval on Retrieval-Augmented Language Modelscited by 8
  27. 2023Backward Compatibility During Data Updates by Weight Interpolationcited by 7
  28. 2025Less LLM, More Documents: Searching for Improved RAGcited by 4
  29. 2026Agentic Search in the Wild: Intents and Trajectory Dynamics from 14M+ Real Search Requestscited by 4
  30. 2025RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentationcited by 3
  31. 2026Beyond Parallel Sampling: Diverse Query Initialization for Agentic Searchcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecastscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Hierarchical Consistency Framework for Auditing Retrieval-Augmented Generation Systems
  2. 2025Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-k
  3. 2026Does Faithfulness-Guided Alignment Hurt Accuracy? Unlocking Accurate and Faithful Post-Retrieval Reasoning
  4. 2025Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Home
  5. 2026Beyond Factual Accuracy: Evaluating Global Reasoning Integrity in RAG Systems with LogicScore
  6. 2026Prompt-Based Abstention Fails Under Misleading Context: A Controlled Study of Small Frozen RAG Models
  7. 2026Why RAGs Hallucinate: Penalty-Aware Evaluation of Retrieval-Augmented Generation Systems with Knowledge-Gap Canaries
  8. 2026A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation