mlsanityData to 13 Oct 2026

Two Axes of LLM Abstention: Answer Correctness and Question Answerability

arXiv 2607.08456 ↗First posted 9 Jul 2026cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

A model should refuse two different things: answers it would get wrong, and questions it should not answer at all, such as unanswerable ones or ones resting on a false premise. The usual recipe thresholds a single confidence score, which cannot tell these apart. Across five instruction-tuned models from three families (2B to 14B), we find they are separate axes. Ordinary answer-confidence tracks whether an answer is right but is nearly blind to whether the question is answerable; a linear probe on hidden states does the reverse. The blind spot does not shrink with scale. It is worst on naturally occurring false-premise questions (CREPE). There, answer-confidence, P(IK), P(True), and even asking the model outright whether a premise is false all stay near chance, while a hidden-state probe reaches 0.69 to 0.77 AUROC: the model represents a problem it will not report. This turns out to be fixable. Instructing a model to check premises backfires, because it then disputes sound and false premises alike (57% false challenges), unable to tell them apart; routing the same instruction with the probe roughly triples challenge precision. We turn the two axes into a calibrated policy that answers only when an answerability score and a correctness score each clear a separately certifies behave differently: the unanswerable-answer rate is controllable at every scale, while the wrong-answer rate is capped by model accuracy, so the guarantee tightens as threshold policy certifies both budgets at 0.75 coverage of correct answers, against 0.31 for a single threshold; at 14B it is the only policy that certifies at all.

Key facts. Cited by 5 papers in the map (data through 13 Oct 2026). Builds on 20 of its 22 references that are in the map. First posted on arXiv 9 Jul 2026.

Builds on

20 of 22 references.

References · 20

  1. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,371
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,823
  3. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 983
  4. 2023The Internal State of an LLM Knows When It's Lyingcited by 652
  5. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 321
  6. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 263
  7. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 255
  8. 2023Do Large Language Models Know What They Don't Know?cited by 240
  9. 2020Logic Tensor Networkscited by 197
  10. 2024Language Models with Conformal Factuality Guaranteescited by 131
  11. 2024Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Taskscited by 78
  12. 2022CREPE: Open-Domain Question Answering with False Presuppositionscited by 71
8 more
  1. 2024Mitigating LLM Hallucinations via Conformal Abstentioncited by 58
  2. 2023The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Modelscited by 55
  3. 2025No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probescited by 37
  4. 2025Query-Level Uncertainty in Large Language Modelscited by 14
  5. 2025Detecting (Un)answerability in Large Language Models with Linear Directionscited by 12
  6. 2026Entropy Alone is Insufficient for Safe Selective Prediction in LLMscited by 7
  7. 2026Are LLM Uncertainty and Correctness Encoded by the Same Features? A Functional Dissociation via Sparse Autoencoderscited by 5
  8. 2021Neural-Symbolic Integration for Interactive Learning and Conceptual Groundingcited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Don't `Well, Actually' Me Unless You Know What You're Talking About: Weak Presupposition Verification Degrades General QA Performancecited by 1
  2. 2026ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signalscited by 1
  3. 2026One Score, Two Decisions: Selective Prediction on the Rare-Disease Tailcited by 0
  4. 2026When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenescited by 0
  5. 2026Have I Seen Enough? Frozen Video-Language Models Encode Evidence Readinesscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models
  2. 2025Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident, Especially When They are Wrong
  3. 2026Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Models
  4. 2024Confidence in the Reasoning of Large Language Models
  5. 2026Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy
  6. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification
  7. 2026CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
  8. 2026Calibrated Confidence Estimation for Tabular Question Answering