mlsanityData to 7 Oct 2026

Neural Text Generation with Unlikelihood Training

arXiv 1908.04319 ↗First posted 12 Aug 2019cs.LG cs.CL stat.MLcited by 489

Not on the map. Why?

Cited by 37 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and decoding leads to dull and repetitive outputs. While some post-hoc fixes have been proposed, in particular top-k and nucleus sampling, they do not address the fact that the token-level probabilities predicted by the model are poor. In this paper we show that the likelihood objective itself is at fault, resulting in a model that assigns too much probability to sequences containing repeats and frequent words, unlike those from the human training distribution. We propose a new objective, unlikelihood training, which forces unlikely generations to be assigned lower probability by the model. We show that both token and sequence level unlikelihood training give less repetitive, less dull text while maintaining perplexity, giving superior generations using standard greedy or beam search. According to human evaluations, our approach with standard beam search also outperforms the currently popular decoding methods of nucleus sampling or beam blocking, thus providing a strong alternative to existing techniques.

Key facts. Cited by 489 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 2; 2021: 68; 2022: 95; 2023: 98; 2024: 86; 2025: 52; 2026: 87.

489 papers.

Sort:

By area:

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  6. 2022Red Teaming Language Models with Language Modelscited by 983
  7. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  8. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  9. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  10. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  11. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  12. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
38 more
  1. 2023Augmented Language Models: a Surveycited by 398
  2. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  3. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  4. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  5. 2023Challenges and Applications of Large Language Modelscited by 251
  6. 2023Pretraining Language Models with Human Preferencescited by 245
  7. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  8. 2022A Contrastive Framework for Neural Text Generationcited by 224
  9. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  10. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  11. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Controlcited by 191
  12. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  13. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  14. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  15. 2023On the Reliability of Watermarks for Large Language Modelscited by 170
  16. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  17. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 153
  18. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  19. 2023Controlled Decoding from Language Modelscited by 133
  20. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  21. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  22. 2022Natural Language to Code Translation with Executioncited by 122
  23. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 122
  24. 2024Direct Preference Optimization with an Offsetcited by 122
  25. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  26. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  27. 2022Locally Typical Samplingcited by 108
  28. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  29. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  30. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  31. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  32. 2023WebCPM: Interactive Web Search for Chinese Long-form Question Answeringcited by 93
  33. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  34. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 90
  35. 2022On Decoding Strategies for Neural Text Generatorscited by 89
  36. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  37. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  38. 2023System 2 Attention (is something you might need too)cited by 85

50 most cited of 489 All 489 (JSON)