# Truncation Sampling as Language Model Desmoothing

arXiv: https://arxiv.org/abs/2210.15191 · HTML: https://mlsanity.com/paper/2210.15191/

## Key facts

- Cited by 123 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 27 Oct 2022.

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [Automatic Detection of Generated Text is Easiest when Humans are Fooled](https://mlsanity.com/paper/1911.00650/) (2019)
- [Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement](https://mlsanity.com/paper/1903.06059/) (2019)
- [Sparse Sequence-to-Sequence Models](https://mlsanity.com/paper/1905.05702/) (2019)
- [Unifying Human and Statistical Evaluation for Natural Language Generation](https://mlsanity.com/paper/1904.02792/) (2019)
- [Epistemic Neural Networks](https://mlsanity.com/paper/2107.08924/) (2021)
- [Language Model Evaluation Beyond Perplexity](https://mlsanity.com/paper/2106.00085/) (2021)
- [Improved Natural Language Generation via Loss Truncation](https://mlsanity.com/paper/2004.14589/) (2020)
- [Revisiting the Uniform Information Density Hypothesis](https://mlsanity.com/paper/2109.11635/) (2021)
- [Consistency of a Recurrent Language Model With Respect to Incomplete Decoding](https://mlsanity.com/paper/2002.02492/) (2020)
- [A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation](https://mlsanity.com/paper/2009.07243/) (2020)
- [On the probability-quality paradox in language generation](https://mlsanity.com/paper/2203.17217/) (2022)
- [A Cognitive Regularizer for Language Modeling](https://mlsanity.com/paper/2105.07144/) (2021)
- [Conditional Poisson Stochastic Beam Search](https://mlsanity.com/paper/2109.11034/) (2021)
- [An Analysis of the Effects of Decoding Algorithms on Fairness in Open-Ended Language Generation](https://mlsanity.com/paper/2210.03826/) (2022)
- [Speeding Up Entmax](https://mlsanity.com/paper/2111.06832/) (2021)

## Built on by (most cited first, 30 of 123)

- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors](https://mlsanity.com/paper/2405.07940/) (2024)
- [Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs](https://mlsanity.com/paper/2407.01082/) (2024)
- [From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models](https://mlsanity.com/paper/2406.16838/) (2024)
- [A Thorough Examination of Decoding Methods in the Era of LLMs](https://mlsanity.com/paper/2402.06925/) (2024)
- [Evaluating the World Model Implicit in a Generative Model](https://mlsanity.com/paper/2406.03689/) (2024)
- [Evaluating Large Language Models on Controlled Generation Tasks](https://mlsanity.com/paper/2310.14542/) (2023)
- [Applying Large Language Models and Chain-of-Thought for Automatic Scoring](https://mlsanity.com/paper/2312.03748/) (2023)
- [Uncertainty in Natural Language Generation: From Theory to Applications](https://mlsanity.com/paper/2307.15703/) (2023)
- [Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation](https://mlsanity.com/paper/2305.09860/) (2023)
- [Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Sampling](https://mlsanity.com/paper/2408.17355/) (2024)
- [It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk](https://mlsanity.com/paper/2310.01387/) (2023)
- [Embarrassingly Simple Self-Distillation Improves Code Generation](https://mlsanity.com/paper/2604.01193/) (2026)
- [Recurrent Drafter for Fast Speculative Decoding in Large Language Models](https://mlsanity.com/paper/2403.09919/) (2024)
- [MAUVE Scores for Generative Models: Theory and Practice](https://mlsanity.com/paper/2212.14578/) (2022)
- [Benchmarking and Improving Text-to-SQL Generation under Ambiguity](https://mlsanity.com/paper/2310.13659/) (2023)
- [Faster Minimum Bayes Risk Decoding with Confidence-based Pruning](https://mlsanity.com/paper/2311.14919/) (2023)
- [Conformal Nucleus Sampling](https://mlsanity.com/paper/2305.02633/) (2023)
- [MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods](https://mlsanity.com/paper/2309.10966/) (2023)
- [Closing the Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/2310.01693/) (2023)
- [LLM Probability Concentration: How Alignment Shrinks the Generative Horizon](https://mlsanity.com/paper/2506.17871/) (2025)
- [Linear-time Minimum Bayes Risk Decoding with Reference Aggregation](https://mlsanity.com/paper/2402.04251/) (2024)
- [Improving Open-Ended Text Generation via Adaptive Decoding](https://mlsanity.com/paper/2402.18223/) (2024)
- [Stealing the Decoding Algorithms of Language Models](https://mlsanity.com/paper/2303.04729/) (2023)
- [Balancing Diversity and Risk in LLM Sampling: How to Select Your Method and Parameter for Open-Ended Text Generation](https://mlsanity.com/paper/2408.13586/) (2024)
- [On the Efficacy of Sampling Adapters](https://mlsanity.com/paper/2307.03749/) (2023)
- [A Frustratingly Simple Decoding Method for Neural Text Generation](https://mlsanity.com/paper/2305.12675/) (2023)
- [MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies](https://mlsanity.com/paper/2305.16958/) (2023)
- [Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2401.02749/) (2024)
- [Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputs](https://mlsanity.com/paper/2510.01218/) (2025)

## Abstract (arXiv)

Long samples of text from neural language models can be of poor quality. Truncation sampling algorithms--like top-p or top-k -- address this by setting some words' probabilities to zero at each step. This work provides framing for the aim of truncation, and an improved algorithm for that aim. We propose thinking of a neural language model as a mixture of a true distribution and a smoothing distribution that avoids infinite perplexity. In this light, truncation algorithms aim to perform desmoothing, estimating a subset of the support of the true distribution. Finding a good subset is crucial: we show that top-p unnecessarily truncates high-probability words, for example causing it to truncate all words but Trump for a document that starts with Donald. We introduce η-sampling, which truncates words below an entropy-dependent probability threshold. Compared to previous algorithms, η-sampling generates more plausible long English documents according to humans, is better at breaking out of repetition, and behaves more reasonably on a battery of test distributions.