# Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs

arXiv: https://arxiv.org/abs/2511.04869 · HTML: https://mlsanity.com/paper/2511.04869/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2025.

## Built on by (most cited first, 19 of 19)

- [Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning](https://mlsanity.com/paper/2512.19920/) (2025)
- [Large Language Models Are Overconfident in Their Own Responses](https://mlsanity.com/paper/2606.03437/) (2026)
- [Hallucinations Undermine Trust; Metacognition is a Way Forward](https://mlsanity.com/paper/2605.01428/) (2026)
- [MedClarify: An information-seeking AI agent for medical diagnosis with case-specific follow-up questions](https://mlsanity.com/paper/2602.17308/) (2026)
- [Semantic Calibration Prevails Where Token Confidence Fails: Benchmarking Long-Form Scientific QA](https://mlsanity.com/paper/2602.00279/) (2026)
- [The Well-Tempered Classifier: Some Elementary Properties of Temperature Scaling](https://mlsanity.com/paper/2602.14862/) (2026)
- [RLTL;DR: Self-improvement by Internalizing Self-generated Feedback](https://mlsanity.com/paper/2609.37633/) (2026)
- [Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models](https://mlsanity.com/paper/2603.26434/) (2026)
- [Task-Aware Calibration: Provably Optimal Decoding in LLMs](https://mlsanity.com/paper/2605.10202/) (2026)
- [Discovery of Hidden Miscalibration Regimes](https://mlsanity.com/paper/2605.13484/) (2026)
- [Concretized Proposition Prompting Resolves Composition-Knowledge Dichotomy in Large Language Models](https://mlsanity.com/paper/2607.08018/) (2026)
- [Auto-Fill: Learning to Predict Missing Values Accurately with Specialist Language Models](https://mlsanity.com/paper/2607.19847/) (2026)
- [What Process Evaluation of Coding Agents Actually Measures: Action, Task, and Step Are Three Different Levels](https://mlsanity.com/paper/2608.22960/) (2026)
- [Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores](https://mlsanity.com/paper/2608.31068/) (2026)
- [Stochastic Semantic Evidence Graphs: Uncertainty Propagation and Governance for Agentic AI](https://mlsanity.com/paper/2609.29703/) (2026)
- [Guard Models Are Overconfident Where Base Models Are Uncertain](https://mlsanity.com/paper/2609.36477/) (2026)
- [CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inference](https://mlsanity.com/paper/2609.36689/) (2026)
- [Verbalized and Internal Probabilities Are Coupled in Large Language Models](https://mlsanity.com/paper/2610.00827/) (2026)
- [Can LLM Agents Efficiently Identify Appliance Faults? Insights From Diagnosing a Simulated Appliance](https://mlsanity.com/paper/2610.19291/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) often lack meaningful confidence estimates for their outputs. While base LLMs are known to exhibit next-token calibration, it remains unclear whether they can assess confidence in the actual meaning of their responses beyond the token level. We find that, when using a certain sampling-based notion of semantic calibration, base LLMs are remarkably well-calibrated: they can meaningfully assess confidence in open-domain question-answering tasks, despite not being explicitly trained to do so. Our main theoretical contribution establishes a mechanism for why semantic calibration emerges as a byproduct of next-token prediction, leveraging a recent connection between calibration and local loss optimality. The theory relies on a general definition of "B-calibration," which is a notion of calibration parameterized by a choice of equivalence classes (semantic or otherwise). This theoretical mechanism leads to a testable prediction: base LLMs will be semantically calibrated when they can easily predict their own distribution over semantic answer classes before generating a response. We state three implications of this prediction, which we validate through experiments: (1) Base LLMs are semantically calibrated across question-answering tasks, (2) RL instruction-tuning systematically breaks this calibration, and (3) chain-of-thought reasoning breaks calibration. To our knowledge, our work provides the first principled explanation of when and why semantic calibration emerges in LLMs.