mlsanityData to 7 Oct 2026

The Curious Case of Neural Text Degeneration

arXiv 1904.09751 ↗First posted 22 Apr 2019cs.CLcited by 2,881

Not on the map. Why?

Cited by 232 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive. In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model. Our findings motivate Nucleus Sampling, a simple but effective method to draw the best out of neural generation. By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.

Key facts. Cited by 2,881 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 3; 2021: 274; 2022: 374; 2023: 564; 2024: 599; 2025: 547; 2026: 520.

2,881 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2023A Survey of Large Language Modelscited by 2,537
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  11. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
38 more
  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  5. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  6. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  7. 2023LIMA: Less Is More for Alignmentcited by 1,080
  8. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  9. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  10. 2022Red Teaming Language Models with Language Modelscited by 983
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  12. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  13. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  14. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  15. 2025Large Language Diffusion Modelscited by 872
  16. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  17. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  18. 2023StarCoder: may the source be with you!cited by 790
  19. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  20. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  21. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  22. 2022A Survey on In-context Learningcited by 717
  23. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  24. 2024Chronos: Learning the Language of Time Seriescited by 712
  25. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  26. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  27. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  28. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  29. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  30. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  31. 2024A Survey on Large Language Models for Code Generationcited by 646
  32. 2022PAL: Program-aided Language Modelscited by 635
  33. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  34. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  35. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  36. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  37. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  38. 2023Enabling Large Language Models to Generate Text with Citationscited by 584

50 most cited of 2,881 All 2,881 (JSON)