# Calibration, Entropy Rates, and Memory in Language Models

arXiv: https://arxiv.org/abs/1906.05664 · HTML: https://mlsanity.com/paper/1906.05664/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2019.

## Built on by (most cited first, 30 of 32)

- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [GPT-NER: Named Entity Recognition via Large Language Models](https://mlsanity.com/paper/2304.10428/) (2023)
- [Calibrated Language Models Must Hallucinate](https://mlsanity.com/paper/2311.14648/) (2023)
- [Locally Typical Sampling](https://mlsanity.com/paper/2202.00666/) (2022)
- [InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation](https://mlsanity.com/paper/2112.01589/) (2021)
- [Mitigating Biases for Instruction-following Language Models via Bias Neurons Elimination](https://mlsanity.com/paper/2311.09627/) (2023)
- [Generative learning for nonlinear dynamics](https://mlsanity.com/paper/2311.04128/) (2023)
- [A Study on the Calibration of In-context Learning](https://mlsanity.com/paper/2312.04021/) (2023)
- [On the Efficacy of Sampling Adapters](https://mlsanity.com/paper/2307.03749/) (2023)
- [Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression](https://mlsanity.com/paper/2310.11428/) (2023)
- [High dimensional online calibration in polynomial time](https://mlsanity.com/paper/2504.09096/) (2025)
- [Probing BERT's priors with serial reproduction chains](https://mlsanity.com/paper/2202.12226/) (2022)
- [Rip van Winkle's Razor: A Simple Estimate of Overfit to Test Data](https://mlsanity.com/paper/2102.13189/) (2021)
- [Language Model Decoding as Direct Metrics Optimization](https://mlsanity.com/paper/2310.01041/) (2023)
- [FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees](https://mlsanity.com/paper/2411.02603/) (2024)
- [UNCERTAINTY-LINE: Length-Invariant Estimation of Uncertainty for Large Language Models](https://mlsanity.com/paper/2505.19060/) (2025)
- [Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review](https://mlsanity.com/paper/2504.18346/) (2025)
- [On the Entropy Calibration of Language Models](https://mlsanity.com/paper/2511.11966/) (2025)
- [CONTESTS: a Framework for Consistency Testing of Span Probabilities in Language Models](https://mlsanity.com/paper/2409.19984/) (2024)
- [Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models](https://mlsanity.com/paper/2603.12248/) (2026)
- [Unsupervised Calibration through Prior Adaptation for Text Classification using Large Language Models](https://mlsanity.com/paper/2307.06713/) (2023)
- [Entropy-Aligned Decoding of LMs for Better Writing and Reasoning](https://mlsanity.com/paper/2601.01714/) (2026)
- [RETO: A Rotary-Enhanced Transformer Operator for High-Fidelity Prediction of Automotive Aerodynamics](https://mlsanity.com/paper/2605.00062/) (2026)
- [Repeated Sequences Reveal Gaps between Large Language Models and Natural Language](https://mlsanity.com/paper/2605.24850/) (2026)
- [Local Grammar-Based Coding Revisited](https://mlsanity.com/paper/2209.13636/) (2022)
- [Extracting Victim Counts from Text](https://mlsanity.com/paper/2302.12367/) (2023)
- [Advancing sleep detection by modelling weak label sets: A novel weakly supervised learning approach](https://mlsanity.com/paper/2402.17601/) (2024)
- [Extracting Disaster Impacts and Impact Related Locations in Social Media Posts Using Large Language Models](https://mlsanity.com/paper/2511.21753/) (2025)
- [Entropy and Attention Dynamics in Small Language Models: A Trace-Level Structural Analysis on the TruthfulQA Benchmark](https://mlsanity.com/paper/2604.03589/) (2026)

## Abstract (arXiv)

Building accurate language models that capture meaningful long-term dependencies is a core challenge in natural language processing. Towards this end, we present a calibration-based approach to measure long-term discrepancies between a generative sequence model and the true distribution, and use these discrepancies to improve the model. Empirically, we show that state-of-the-art language models, including LSTMs and Transformers, are miscalibrated: the entropy rates of their generations drift dramatically upward over time. We then provide provable methods to mitigate this phenomenon. Furthermore, we show how this calibration-based approach can also be used to measure the amount of memory that language models use for prediction.