# Evaluating Distributional Distortion in Neural Language Modeling

arXiv: https://arxiv.org/abs/2203.12788 · HTML: https://mlsanity.com/paper/2203.12788/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 24 Mar 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020)
- [Distributionally Robust Language Modeling](https://mlsanity.com/paper/1909.02060/) (2019)
- [How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN](https://mlsanity.com/paper/2111.09509/) (2021)
- [On NMT Search Errors and Model Errors: Cat Got Your Tongue?](https://mlsanity.com/paper/1908.10090/) (2019)
- [The Devil is in the Tails: Fine-grained Classification in the Wild](https://mlsanity.com/paper/1709.01450/) (2017)
- [Language Model Evaluation Beyond Perplexity](https://mlsanity.com/paper/2106.00085/) (2021)
- [Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoders](https://mlsanity.com/paper/2005.02590/) (2020)
- [Consistency of a Recurrent Language Model With Respect to Incomplete Decoding](https://mlsanity.com/paper/2002.02492/) (2020)
- [Factors in Finetuning Deep Model for object detection](https://mlsanity.com/paper/1601.05150/) (2016)
- [Examining the Inductive Bias of Neural Language Models with Artificial Languages](https://mlsanity.com/paper/2106.01044/) (2021)
- [Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level](https://mlsanity.com/paper/2105.06020/) (2021)
- [Globally Normalized Transition-Based Neural Networks](https://mlsanity.com/paper/1603.06042/) (2016)
- [Evaluating Computational Language Models with Scaling Properties of Natural Language](https://mlsanity.com/paper/1906.09379/) (2019)
- [A Tale Of Two Long Tails](https://mlsanity.com/paper/2107.13098/) (2021)
- [On Language Models for Creoles](https://mlsanity.com/paper/2109.06074/) (2021)
- [Do Neural Nets Learn Statistical Laws behind Natural Language?](https://mlsanity.com/paper/1707.04848/) (2017)
- [Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction](https://mlsanity.com/paper/1909.02855/) (2019)
- [An Empirical Investigation of Global and Local Normalization for Recurrent Neural Sequence Models Using a Continuous Relaxation to Beam Search](https://mlsanity.com/paper/1904.06834/) (2019)
- [The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers](https://mlsanity.com/paper/2010.08127/) (2020)
- [Are Some Words Worth More than Others?](https://mlsanity.com/paper/2010.06069/) (2020)
- [Non-Parametric Few-Shot Learning for Word Sense Disambiguation](https://mlsanity.com/paper/2104.12677/) (2021)
- [Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models](https://mlsanity.com/paper/2004.14601/) (2020)
- [Mode recovery in neural autoregressive sequence modeling](https://mlsanity.com/paper/2106.05459/) (2021)

## Built on by (most cited first, 19 of 19)

- [Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning](https://mlsanity.com/paper/2301.11916/) (2023)
- [A Tale of Tails: Model Collapse as a Change of Scaling Laws](https://mlsanity.com/paper/2402.07043/) (2024)
- [The Best Instruction-Tuning Data are Those That Fit](https://mlsanity.com/paper/2502.04194/) (2025)
- [Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification](https://mlsanity.com/paper/2406.07515/) (2024)
- [Emergent Representations of Program Semantics in Language Models Trained on Programs](https://mlsanity.com/paper/2305.11169/) (2023)
- [Tailoring Language Generation Models under Total Variation Distance](https://mlsanity.com/paper/2302.13344/) (2023)
- [Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics](https://mlsanity.com/paper/2209.10015/) (2022)
- [LLM as a Broken Telephone: Iterative Generation Distorts Information](https://mlsanity.com/paper/2502.20258/) (2025)
- [Why Less is More (Sometimes): A Theory of Data Curation](https://mlsanity.com/paper/2511.03492/) (2025)
- [EMO: Earth Mover Distance Optimization for Auto-Regressive Language Modeling](https://mlsanity.com/paper/2310.04691/) (2023)
- [ModelShield: Adaptive and Robust Watermark against Model Extraction Attack](https://mlsanity.com/paper/2405.02365/) (2024)
- [Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory](https://mlsanity.com/paper/2410.08942/) (2024)
- [Predict the Next Word: Humans exhibit uncertainty in this task and language models _____](https://mlsanity.com/paper/2402.17527/) (2024)
- [Neural-Symbolic Inference for Robust Autoregressive Graph Parsing via Compositional Uncertainty Quantification](https://mlsanity.com/paper/2301.11459/) (2023)
- [FLAMES: Improving LLM Math Reasoning via a Fine-Grained Analysis of the Data Synthesis Pipeline](https://mlsanity.com/paper/2508.16514/) (2025)
- [On Using Distribution-Based Compositionality Assessment to Evaluate Compositional Generalisation in Machine Translation](https://mlsanity.com/paper/2311.08249/) (2023)
- [Relative Overfitting and Accept-Reject Framework](https://mlsanity.com/paper/2505.07783/) (2025)
- [Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance](https://mlsanity.com/paper/2604.17650/) (2026)
- [When transformers learn "impossible" languages, what do they learn?](https://mlsanity.com/paper/2606.30815/) (2026)

## Abstract (arXiv)

A fundamental characteristic of natural language is the high rate at which speakers produce novel expressions. Because of this novelty, a heavy-tail of rare events accounts for a significant amount of the total probability mass of distributions in language (Baayen, 2001). Standard language modeling metrics such as perplexity quantify the performance of language models (LM) in aggregate. As a result, we have relatively little understanding of whether neural LMs accurately estimate the probability of sequences in this heavy-tail of rare events. To address this gap, we develop a controlled evaluation scheme which uses generative models trained on natural data as artificial languages from which we can exactly compute sequence probabilities. Training LMs on generations from these artificial languages, we compare the sequence-level probability estimates given by LMs to the true probabilities in the target language. Our experiments reveal that LSTM and Transformer language models (i) systematically underestimate the probability of sequences drawn from the target language, and (ii) do so more severely for less-probable sequences. Investigating where this probability mass went, (iii) we find that LMs tend to overestimate the probability of ill formed (perturbed) sequences. In addition, we find that this underestimation behaviour (iv) is weakened, but not eliminated by greater amounts of training data, and (v) is exacerbated for target distributions with lower entropy.