mlsanityData to 7 Oct 2026

Before Answering: Evidence Sufficiency under Size-Matched Memory Construction

arXiv 2609.32269 ↗First posted 26 Sep 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Agents that answer questions from compressed or retrieved memory must recognize when the evidence a query needs is no longer in memory. Benchmarks for this task usually create insufficient-evidence examples by deleting supporting passages. We show that this construction leaks the label through memory size: on MuSiQue, a classifier that only counts paragraphs reaches an area under the ROC curve (AUROC) of 0.979 for detecting unsafe memory, higher than the lexical estimator we initially evaluated. We propose a size-matched construction that provably removes this shortcut, and use it to study MemSafe, an estimator that cross-encodes the query with each memory unit and aggregates the units with a set transformer. Across three multi-hop question answering datasets and five seeds, MemSafe reaches 0.968 and 0.983 AUROC on MuSiQue and HotpotQA, 0.26 to 0.39 above a lexical baseline, while the third dataset, 2WikiMultiHopQA, is saturated. A frozen pretrained cross-encoder with a logistic head already closes 41% of the MuSiQue gap between the lexical baseline and MemSafe. At the same time, MemSafe degrades more than a weak baseline on the unanswerable questions released with MuSiQue, reaches only 0.639 AUROC on SQuAD~2.0, and needs several thousand clinical training examples before it outperforms a feature-based estimator. Used as a gate for a 7B reader, it reduces the error rate on answered questions from 0.850 to 0.631 at 5% coverage, outperforming both reader confidence and, on average, the ground-truth integrity label, although a 7B LLM judge is the better gate at 10% coverage. These results indicate that the way insufficient evidence is constructed matters as much as the estimator that detects it.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 45 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

38 of 45 references.

References · 38

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  6. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  8. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  9. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  10. 2017Selective Classification for Deep Neural Networkscited by 799
  11. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  12. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
26 more
  1. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
  2. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  3. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 256
  4. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  5. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  6. 2025How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behaviorcited by 95
  7. 2025MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agentscited by 82
  8. 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 75
  9. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 69
  10. 2024CompAct: Compressing Retrieved Documents Actively for Question Answeringcited by 64
  11. 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Modelscited by 55
  12. 2025Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Homecited by 33
  13. 2024Probing-RAG: Self-Probing to Guide Language Models in Selective Document Retrievalcited by 28
  14. 2023Plugin estimators for selective classification with out-of-distribution detectioncited by 14
  15. 2024RetrievalQA: Assessing Adaptive Retrieval-Augmented Generation for Short-form Open-Domain Question Answeringcited by 13
  16. 2025Detecting (Un)answerability in Large Language Models with Linear Directionscited by 12
  17. 2026S2G-RAG: Structured Sufficiency and Gap Judging for Iterative Retrieval-Augmented QAcited by 9
  18. 2023Fair Classifiers that Abstain without Harmcited by 8
  19. 2025LLM-Independent Adaptive RAG: Let the Question Speak for Itselfcited by 7
  20. 2025SAFER: Risk-Constrained Sample-then-Filter in Large Language Modelscited by 7
  21. 2026SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generationcited by 6
  22. 2025Know When to Abstain: Optimal Selective Classification with Likelihood Ratioscited by 5
  23. 2026Failure Modes in Multi-Hop QA: The Weakest Link Effect and the Recognition Bottleneckcited by 3
  24. 2026RASER: Recoverability-Aware Selective Escalation Router for Multi-Hop Question Answeringcited by 2
  25. 2026Evidence Absence Is Not Evidence Insufficiency: Diagnosing NEI Construction Artifacts in Fact Verificationcited by 1
  26. 2026EvoTrustRAG: Evolution-Aware Conflict Attribution and Evidence Handling for Reliable Retrieval-Augmented Generationcited by 1

Built on by

Not cited yet.