mlsanityData to 14 Oct 2026

Why RAGs Hallucinate: Penalty-Aware Evaluation of Retrieval-Augmented Generation Systems with Knowledge-Gap Canaries

arXiv 2608.26385 ↗First posted 26 Aug 2026cs.CL cs.AIcited by 0

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Volume-based accuracy rewards retrieval-augmented generation (RAG) systems for guessing: a system that answers everything outscores one that declines when its knowledge base cannot support an answer. Building on the confidence-target analysis of Kalai et al. (2025), we present a penalty-aware evaluation framework for deployed RAG products, combining (i) asymmetric scoring (correct +1, wrong -4, abstain 0), (ii) knowledge-gap canaries, questions whose answers are verifiably absent from the knowledge base, so that any answer constitutes ungrounded generation from parametric memory, and (iii) a failure-attribution pipeline that separates retrieval, generation, and abstention-policy failures. Applying the framework to three commercial RAG systems and a no-retrieval baseline on SimpleQA-Verified (1,000 questions x 3 repeats, graded blind by a cross-family three-judge panel with 98.9% unanimity), we find that accuracy when answering is closely clustered across systems (97.0-98.0%), while canary violation rates differ roughly sixfold (16.7% vs. 98.1%). The systems are separated less by what they answer correctly than by whether they answer at all when they should not, and penalty-aware scoring reorders the volume-based ranking accordingly; the reordering is stable across penalty settings from k=1 to k=9. All code, configurations, transcripts, and judge votes are released for independent audit.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 12 of its 13 references that are in the map. First posted on arXiv 26 Aug 2026.

Builds on

12 of 13 references.

References · 12

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  6. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  7. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 418
  8. 2024Measuring short-form factuality in large language modelscited by 373
  9. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 341
  10. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
  11. 2025Why Language Models Hallucinatecited by 220
  12. 2025SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledgecited by 54

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AB-RAG: Adaptive Budgeted Retrieval-Augmented Generation for Reliable Question Answering
  2. 2024Semantic Tokens in Retrieval Augmented Generation
  3. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  4. 2026A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation
  5. 2025Bidirectional RAG: Safe Self-Improving Retrieval-Augmented Generation Through Multi-Stage Validation
  6. 2026When Confidence Takes the Wrong Path: Diagnosing Retrieval-State Lock-In in RAG
  7. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Level
  8. 2026RAGStress: A controlled benchmark for evaluating retrieval-augmented generation under knowledge-base degradation