mlsanityData to 7 Oct 2026

Locally Typical Sampling

arXiv 2202.00666 ↗First posted 1 Feb 2022cs.CL cs.AIcited by 108

Not on the map. Why?

Abstract (arXiv)

Today's probabilistic language generators fall short when it comes to producing coherent and fluent text despite the fact that the underlying models perform well under standard metrics, e.g., perplexity. This discrepancy has puzzled the language generation community for the last few years. In this work, we posit that the abstraction of natural language generation as a discrete stochastic process--which allows for an information-theoretic analysis--can provide new insights into the behavior of probabilistic language generators, e.g., why high-probability texts can be dull or repetitive. Humans use language as a means of communicating information, aiming to do so in a simultaneously efficient and error-minimizing manner; in fact, psycholinguistics research suggests humans choose each word in a string with this subconscious goal in mind. We formally define the set of strings that meet this criterion: those for which each word has an information content close to the expected information content, i.e., the conditional entropy of our model. We then propose a simple and efficient procedure for enforcing this criterion when generating from probabilistic models, which we call locally typical sampling. Automatic and human evaluations show that, in comparison to nucleus and top-k sampling, locally typical sampling offers competitive performance (in both abstractive summarization and story generation) in terms of quality while consistently reducing degenerate repetitions.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Builds on 28 of its 65 references that are in the map. First posted on arXiv 1 Feb 2022.

Builds on

28 of 65 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  6. 2018Hierarchical Neural Story Generationcited by 1,182
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  8. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  9. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  10. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  11. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  12. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
16 more
  1. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 137
  2. 2020A Knowledge-Enhanced Pretraining Model for Commonsense Story Generationcited by 116
  3. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
  4. 2020If beam search is the answer, what was the question?cited by 114
  5. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  6. 2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
  7. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 94
  8. 2019Do Massively Pretrained Language Models Make Better Storytellers?cited by 90
  9. 2020Generalized Entropy Regularization or: There's Nothing Special about Label Smoothingcited by 40
  10. 2019Calibration, Entropy Rates, and Memory in Language Modelscited by 32
  11. 2020A Systematic Characterization of Sampling Algorithms for Open-ended Language Generationcited by 28
  12. 2020Phonotactic Complexity and its Trade-offscited by 23
  13. 2021A surprisal--duration trade-off across and within the world's languagescited by 22
  14. 2020Decoding Methods for Neural Narrative Generationcited by 18
  15. 2019An Empirical Investigation of Global and Local Normalization for Recurrent Neural Sequence Models Using a Continuous Relaxation to Beam Searchcited by 12
  16. 2020Mirostat: A Neural Text Decoding Algorithm that Directly Controls Perplexitycited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 28; 2024: 14; 2025: 25; 2026: 33.

108 papers.

Sort:

By area:

  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  2. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 196
  3. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Controlcited by 191
  4. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  5. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  6. 2023Controlled Text Generation with Natural Language Instructionscited by 87
  7. 2022Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methodscited by 75
  8. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  9. 2023Contrastive Decoding Improves Reasoning in Large Language Modelscited by 71
  10. 2023Applying Large Language Models and Chain-of-Thought for Automatic Scoringcited by 68
  11. 2023Bag of Tricks for Training Data Extraction from Language Modelscited by 67
  12. 2023Selectively Answering Ambiguous Questionscited by 63
38 more
  1. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
  2. 2024Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Samplingcited by 46
  3. 2023Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Modelscited by 40
  4. 2023Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspectivecited by 39
  5. 2023The Consensus Game: Language Model Generation via Equilibrium Searchcited by 35
  6. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 34
  7. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  8. 2025Optimizing Temperature for Language Models with Multi-Sample Inferencecited by 33
  9. 2022A Measure-Theoretic Characterization of Tight Language Modelscited by 32
  10. 2023Closing the Curious Case of Neural Text Degenerationcited by 30
  11. 2023Benchmarking the Generation of Fact Checking Explanationscited by 25
  12. 2024Towards a Scalable Reference-Free Evaluation of Generative Modelscited by 25
  13. 2024Improving Open-Ended Text Generation via Adaptive Decodingcited by 21
  14. 2023Forking Uncertainties: Reliable Prediction and Model Predictive Control with Sequence Models via Conformal Risk Controlcited by 19
  15. 2023G3Detector: General GPT-Generated Text Detectorcited by 17
  16. 2023On the Efficacy of Sampling Adapterscited by 17
  17. 2023MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropiescited by 16
  18. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  19. 2024Multi-Draft Speculative Sampling: Canonical Decomposition and Theoretical Limitscited by 11
  20. 2025Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Modelscited by 11
  21. 2025Foundations of Top-k Decoding For Language Modelscited by 10
  22. 2025InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacycited by 10
  23. 2025Semantic-guided Diverse Decoding for Large Language Modelcited by 9
  24. 2023Transformer Language Models Handle Word Frequency in Prediction Headcited by 8
  25. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  26. 2023The Stable Entropy Hypothesis and Entropy-Aware Decoding: An Analysis and Algorithm for Robust Natural Language Generationcited by 7
  27. 2023EMO: Earth Mover Distance Optimization for Auto-Regressive Language Modelingcited by 7
  28. 2022Best-k Search Algorithm for Neural Text Generationcited by 6
  29. 2023HistAlign: Improving Context Dependency in Language Generation by Aligning with Historycited by 6
  30. 2024Decoding Game: On Minimax Optimality of Heuristic Text Generation Strategiescited by 6
  31. 2022Language Model Decoding as Likelihood-Utility Alignmentcited by 5
  32. 2023Corpus Synthesis for Zero-shot ASR domain Adaptation using Large Language Modelscited by 5
  33. 2024Differentiating Between Human-Written and AI-Generated Texts Using Automatically Extracted Linguistic Featurescited by 5
  34. 2025NavComposer: Composing Language Instructions for Navigation Trajectories through Action-Scene-Object Modularizationcited by 5
  35. 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMscited by 5
  36. 2023Structured Voronoi Samplingcited by 4
  37. 2024Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Modelscited by 4
  38. 2024Unused information in token probability distribution of generative LLM: improving LLM reading comprehension through calculation of expected valuescited by 4

50 most cited of 108 All 108 (JSON)