mlsanityData to 7 Oct 2026

Truncation Sampling as Language Model Desmoothing

arXiv 2210.15191 ↗First posted 27 Oct 2022cs.CLcited by 121

Not on the map. Why?

Abstract (arXiv)

Long samples of text from neural language models can be of poor quality. Truncation sampling algorithms--like top-p or top-k -- address this by setting some words' probabilities to zero at each step. This work provides framing for the aim of truncation, and an improved algorithm for that aim. We propose thinking of a neural language model as a mixture of a true distribution and a smoothing distribution that avoids infinite perplexity. In this light, truncation algorithms aim to perform desmoothing, estimating a subset of the support of the true distribution. Finding a good subset is crucial: we show that top-p unnecessarily truncates high-probability words, for example causing it to truncate all words but Trump for a document that starts with Donald. We introduce η-sampling, which truncates words below an entropy-dependent probability threshold. Compared to previous algorithms, η-sampling generates more plausible long English documents according to humans, is better at breaking out of repetition, and behaves more reasonably on a battery of test distributions.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

25 of 36 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2018Hierarchical Neural Story Generationcited by 1,182
  6. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  7. 2019ELI5: Long Form Question Answeringcited by 567
  8. 2019Text Summarization with Pretrained Encoderscited by 553
  9. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  10. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  11. 2019Automatic Detection of Generated Text is Easiest when Humans are Fooledcited by 260
  12. 2019Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacementcited by 205
13 more
  1. 2019Sparse Sequence-to-Sequence Modelscited by 167
  2. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  3. 2021Epistemic Neural Networkscited by 120
  4. 2021Language Model Evaluation Beyond Perplexitycited by 77
  5. 2020Improved Natural Language Generation via Loss Truncationcited by 75
  6. 2021Revisiting the Uniform Information Density Hypothesiscited by 63
  7. 2020Consistency of a Recurrent Language Model With Respect to Incomplete Decodingcited by 54
  8. 2020A Systematic Characterization of Sampling Algorithms for Open-ended Language Generationcited by 28
  9. 2022On the probability-quality paradox in language generationcited by 16
  10. 2021A Cognitive Regularizer for Language Modelingcited by 11
  11. 2021Conditional Poisson Stochastic Beam Searchcited by 11
  12. 2022An Analysis of the Effects of Decoding Algorithms on Fairness in Open-Ended Language Generationcited by 7
  13. 2021Speeding Up Entmaxcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 21; 2024: 30; 2025: 34; 2026: 34.

121 papers.

Sort:

By area:

  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  2. 2024RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectorscited by 157
  3. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 148
  4. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  5. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 112
  6. 2024Evaluating the World Model Implicit in a Generative Modelcited by 107
  7. 2023Evaluating Large Language Models on Controlled Generation Taskscited by 73
  8. 2023Applying Large Language Models and Chain-of-Thought for Automatic Scoringcited by 68
  9. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  10. 2023Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translationcited by 57
  11. 2023It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Riskcited by 49
  12. 2024Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Samplingcited by 46
38 more
  1. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 36
  2. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Modelscited by 35
  3. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 34
  4. 2023Faster Minimum Bayes Risk Decoding with Confidence-based Pruningcited by 34
  5. 2023MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methodscited by 33
  6. 2023Conformal Nucleus Samplingcited by 32
  7. 2023Benchmarking and Improving Text-to-SQL Generation under Ambiguitycited by 31
  8. 2023Closing the Curious Case of Neural Text Degenerationcited by 30
  9. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  10. 2024Linear-time Minimum Bayes Risk Decoding with Reference Aggregationcited by 21
  11. 2024Improving Open-Ended Text Generation via Adaptive Decodingcited by 21
  12. 2023Stealing the Decoding Algorithms of Language Modelscited by 19
  13. 2024Balancing Diversity and Risk in LLM Sampling: How to Select Your Method and Parameter for Open-Ended Text Generationcited by 18
  14. 2023On the Efficacy of Sampling Adapterscited by 17
  15. 2023A Frustratingly Simple Decoding Method for Neural Text Generationcited by 16
  16. 2023MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropiescited by 16
  17. 2024Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decodingcited by 15
  18. 2025Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputscited by 15
  19. 2023Penalty Decoding: Well Suppress the Self-Reinforcement Effect in Open-Ended Text Generationcited by 13
  20. 2024Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimationcited by 13
  21. 2024Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithmscited by 13
  22. 2024Generating Diverse and High-Quality Texts by Minimum Bayes Risk Decodingcited by 11
  23. 2024REAL Sampling: Boosting Factuality and Diversity of Open-Ended Generation via Asymptotic Entropycited by 11
  24. 2023Look-back Decoding for Open-Ended Text Generationcited by 10
  25. 2023Model-Based Minimum Bayes Risk Decoding for Text Generationcited by 10
  26. 2024Introducing the NewsPaLM MBR and QE Dataset: LLM-Generated High-Quality Parallel Data Outperforms Traditional Web-Crawled Datacited by 10
  27. 2025Uncertainty-Aware Decoding with Minimum Bayes Riskcited by 10
  28. 2025Foundations of Top-k Decoding For Language Modelscited by 10
  29. 2025InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacycited by 10
  30. 2025GUARD: Glocal Uncertainty-Aware Robust Decoding for Effective and Efficient Open-Ended Text Generationcited by 10
  31. 2023Quality-Aware Translation Models: Efficient Generation and Quality Estimation in a Single Modelcited by 9
  32. 2024Don't Throw Away Data: Better Sequence Knowledge Distillationcited by 9
  33. 2025Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Modelscited by 9
  34. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  35. 2025Semantic-guided Diverse Decoding for Large Language Modelcited by 9
  36. 2025Asking a Language Model for Diverse Responsescited by 9
  37. 2024Diversity Explains Inference Scaling Laws: Through a Case Study of Minimum Bayes Risk Decodingcited by 8
  38. 2025Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generationcited by 8

50 most cited of 121 All 121 (JSON)