mlsanityData to 14 Oct 2026

Calibration, Entropy Rates, and Memory in Language Models

arXiv 1906.05664 ↗First posted 11 Jun 2019cs.CL cs.LG stat.MLcited by 32

Not on the map. Why?

Abstract (arXiv)

Building accurate language models that capture meaningful long-term dependencies is a core challenge in natural language processing. Towards this end, we present a calibration-based approach to measure long-term discrepancies between a generative sequence model and the true distribution, and use these discrepancies to improve the model. Empirically, we show that state-of-the-art language models, including LSTMs and Transformers, are miscalibrated: the entropy rates of their generations drift dramatically upward over time. We then provide provable methods to mitigate this phenomenon. Furthermore, we show how this calibration-based approach can also be used to measure the amount of memory that language models use for prediction.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 3; 2023: 11; 2024: 3; 2025: 5; 2026: 7.

32 papers.

Sort:

By area:

  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  2. 2023Cognitive Architectures for Language Agentscited by 429
  3. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 160
  4. 2023Calibrated Language Models Must Hallucinatecited by 115
  5. 2022Locally Typical Samplingcited by 111
  6. 2021InfoLM: A New Metric to Evaluate Summarization & Data2Text Generationcited by 33
  7. 2023Mitigating Biases for Instruction-following Language Models via Bias Neurons Eliminationcited by 23
  8. 2023Generative learning for nonlinear dynamicscited by 22
  9. 2023A Study on the Calibration of In-context Learningcited by 22
  10. 2023On the Efficacy of Sampling Adapterscited by 17
  11. 2023Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregressioncited by 14
  12. 2025High dimensional online calibration in polynomial timecited by 13
20 more
  1. 2022Probing BERT's priors with serial reproduction chainscited by 11
  2. 2021Rip van Winkle's Razor: A Simple Estimate of Overfit to Test Datacited by 8
  3. 2023Language Model Decoding as Direct Metrics Optimizationcited by 8
  4. 2024FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guaranteescited by 7
  5. 2025UNCERTAINTY-LINE: Length-Invariant Estimation of Uncertainty for Large Language Modelscited by 5
  6. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 4
  7. 2025On the Entropy Calibration of Language Modelscited by 4
  8. 2024CONTESTS: a Framework for Consistency Testing of Span Probabilities in Language Modelscited by 3
  9. 2026Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Modelscited by 3
  10. 2023Unsupervised Calibration through Prior Adaptation for Text Classification using Large Language Modelscited by 1
  11. 2026Entropy-Aligned Decoding of LMs for Better Writing and Reasoningcited by 1
  12. 2026RETO: A Rotary-Enhanced Transformer Operator for High-Fidelity Prediction of Automotive Aerodynamicscited by 1
  13. 2026Repeated Sequences Reveal Gaps between Large Language Models and Natural Languagecited by 1
  14. 2022Local Grammar-Based Coding Revisitedcited by 0
  15. 2023Extracting Victim Counts from Textcited by 0
  16. 2024Advancing sleep detection by modelling weak label sets: A novel weakly supervised learning approachcited by 0
  17. 2025Extracting Disaster Impacts and Impact Related Locations in Social Media Posts Using Large Language Modelscited by 0
  18. 2026Entropy and Attention Dynamics in Small Language Models: A Trace-Level Structural Analysis on the TruthfulQA Benchmarkcited by 0
  19. 2026Cognitive Fatigue in Autoregressive Transformers: Formalization and Measurementcited by 0
  20. 2026Identification and Honest Recovery from Semantic Observation Kernels: Operator Error, Coarsening, and Stabilitycited by 0