# Asymptotic Risk Calibration for Selective Question Answering

arXiv: https://arxiv.org/abs/2608.12008 · HTML: https://mlsanity.com/paper/2608.12008/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 41 references that are in the map.
- First posted on arXiv 12 Aug 2026.

## Builds on (most cited first, 19 of 19)

- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [Conformal Language Modeling](https://mlsanity.com/paper/2306.10193/) (2023)
- [Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement](https://mlsanity.com/paper/2407.18370/) (2024)
- [ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guarantees](https://mlsanity.com/paper/2407.00499/) (2024)
- [PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection](https://mlsanity.com/paper/2502.12119/) (2025)
- [Conformal Alignment: Knowing When to Trust Foundation Models with Guarantees](https://mlsanity.com/paper/2405.10301/) (2024)
- [LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering](https://mlsanity.com/paper/2412.12359/) (2024)
- [CoT-Kinetics: A Theoretical Modeling Assessing LRM Reasoning Process](https://mlsanity.com/paper/2505.13408/) (2025)
- [Self-Evolving Multi-Agent Systems via Textual Backpropagation](https://mlsanity.com/paper/2506.09046/) (2025)
- [COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees](https://mlsanity.com/paper/2506.20178/) (2025)
- [Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifiers](https://mlsanity.com/paper/2509.03059/) (2025)
- [OPD-V: Visual On-Policy Self-Distillation with Modality Balance](https://mlsanity.com/paper/2608.05131/) (2026)
- [LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systems](https://mlsanity.com/paper/2512.01556/) (2025)
- [Conformal Lesion Segmentation for 3D Medical Images](https://mlsanity.com/paper/2510.17897/) (2025)
- [Set-Valued Prediction for Large Language Models with Feasibility-Aware Coverage Guarantees](https://mlsanity.com/paper/2603.22966/) (2026)
- [MiRD: Reliable Set-Valued Prediction for Open-Ended Question Answering via Miscoverage Risk Decomposition](https://mlsanity.com/paper/2605.27091/) (2026)

## Abstract (arXiv)

Large language models (LLMs) may generate fluent but incorrect answers, making uncertainty quantification important for reliable question answering. However, heuristic uncertainty scores cannot perfectly distinguish correct predictions from incorrect ones, and directly applying a fixed uncertainty threshold provides no statistical control over the error rate among accepted answers. To address this limitation, we propose A-CRC-QA, a post-hoc calibration framework for uncertainty-aware selective question answering. The proposed method reformulates selection-conditioned error control as a linear expectation constraint and applies a monotonized empirical-risk calibration procedure inspired by conformal risk control. Since the resulting instance-wise loss is generally non-monotone with respect to the acceptance threshold, our framework targets asymptotic rather than finite-sample risk control. A-CRC-QA is model-agnostic, requires no additional training, and can be combined with different uncertainty estimators. Experiments on CoQA and MedMCQA demonstrate its applicability to both open-ended and closed-ended question answering, achieving a favorable trade-off between accepted-answer reliability and answer retention compared with uncalibrated and confidence-bound-based baselines.