# Entropy Alone is Insufficient for Safe Selective Prediction in LLMs

arXiv: https://arxiv.org/abs/2603.21172 · HTML: https://mlsanity.com/paper/2603.21172/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 22 Mar 2026.

## Builds on (most cited first, 14 of 14)

- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search](https://mlsanity.com/paper/2504.08066/) (2025)
- [INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection](https://mlsanity.com/paper/2402.03744/) (2024)
- [Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs](https://mlsanity.com/paper/2406.15927/) (2024)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [Language Models with Conformal Factuality Guarantees](https://mlsanity.com/paper/2402.10978/) (2024)
- [Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers](https://mlsanity.com/paper/1805.08206/) (2018)
- [Real-Time Detection of Hallucinated Entities in Long-Form Generation](https://mlsanity.com/paper/2509.03531/) (2025)
- [Non-Exchangeable Conformal Language Generation with Nearest Neighbors](https://mlsanity.com/paper/2402.00707/) (2024)
- [A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators](https://mlsanity.com/paper/2410.15361/) (2024)
- [Geometric Uncertainty for Detecting and Correcting Hallucinations in LLMs](https://mlsanity.com/paper/2509.13813/) (2025)
- [Fine-Tuning Large Language Models to Appropriately Abstain with Semantic Entropy](https://mlsanity.com/paper/2410.17234/) (2024)
- [TECP: Token-Entropy Conformal Prediction for LLMs](https://mlsanity.com/paper/2509.00461/) (2025)
- [Semantic Self-Distillation for Language Model Uncertainty](https://mlsanity.com/paper/2602.04577/) (2026)

## Built on by (most cited first, 7 of 7)

- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence](https://mlsanity.com/paper/2604.03216/) (2026)
- [Two Axes of LLM Abstention: Answer Correctness and Question Answerability](https://mlsanity.com/paper/2607.08456/) (2026)
- [PassiveQA: A Three-Action Framework for Epistemically Calibrated Question Answering via Supervised Finetuning](https://mlsanity.com/paper/2604.04565/) (2026)
- [Non-Degenerate Risk Certification for Automated Security Decisions: A Decision-Contract Theory with ATT&CK-Aligned Triage as a Worked Instance](https://mlsanity.com/paper/2608.12444/) (2026)
- [No Universal Signal Predicts Sample-Level LLM Regression under Version Updates](https://mlsanity.com/paper/2608.13607/) (2026)
- [From Positionwise Confidence to Prefix Scheduling: Verifier Skipping in Speculative Decoding](https://mlsanity.com/paper/2608.14787/) (2026)

## Abstract (arXiv)

Selective prediction systems can mitigate harms resulting from language model hallucinations by abstaining from answering in high-risk cases. Uncertainty quantification techniques are often employed to identify such cases, but are rarely evaluated in the context of the wider selective prediction policy and its ability to operate at low target error rates. We identify a model-dependent failure mode of entropy-based uncertainty methods that leads to unreliable abstention behaviour, and address it by combining entropy scores with a correctness probe signal. We find that across three QA benchmarks (TriviaQA, BioASQ, MedicalQA) and four model families, the combined score generally improves both the risk--coverage trade-off and calibration performance relative to entropy-only baselines. Our results highlight the importance of deployment-facing evaluation of uncertainty methods, using metrics that directly reflect whether a system can be trusted to operate at a stated risk level.