mlsanityData to 7 Oct 2026

Relating Neural Text Degeneration to Exposure Bias

arXiv 2109.08705 ↗First posted 17 Sep 2021cs.CL cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

This work focuses on relating two mysteries in neural-based text generation: exposure bias, and text degeneration. Despite the long time since exposure bias was mentioned and the numerous studies for its remedy, to our knowledge, its impact on text generation has not yet been verified. Text degeneration is a problem that the widely-used pre-trained language model GPT-2 was recently found to suffer from (Holtzman et al., 2020). Motivated by the unknown causation of the text degeneration, in this paper we attempt to relate these two mysteries. Specifically, we first qualitatively quantitatively identify mistakes made before text degeneration occurs. Then we investigate the significance of the mistakes by inspecting the hidden states in GPT-2. Our results show that text degeneration is likely to be partly caused by exposure bias. We also study the self-reinforcing mechanism of text degeneration, explaining why the mistakes amplify. In sum, our study provides a more concrete foundation for further investigation on exposure bias and text degeneration problems.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 21 of its 40 references that are in the map. First posted on arXiv 17 Sep 2021.

Builds on

21 of 40 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2017Billion-scale similarity search with GPUscited by 2,477
  5. 2019Language Models as Knowledge Bases?cited by 1,922
  6. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  7. 2018Hierarchical Neural Story Generationcited by 1,182
  8. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  9. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  10. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  11. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  12. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
9 more
  1. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 214
  2. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  3. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 126
  4. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  5. 2019Generalization in Generation: A closer look at Exposure Biascited by 123
  6. 2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
  7. 2019Do Massively Pretrained Language Models Make Better Storytellers?cited by 90
  8. 2015How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?cited by 76
  9. 2018Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Modelscited by 69

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 7; 2024: 2; 2025: 3; 2026: 3.

19 papers.

Sort:

By area:

  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  2. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  3. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  4. 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 47
  5. 2023Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspectivecited by 39
  6. 2023ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviewscited by 34
  7. 2022Reward Gaming in Conditional Text Generationcited by 29
  8. 2025Wait, Wait, Wait... Why Do Reasoning Models Loop?cited by 24
  9. 2024The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Modelscited by 17
  10. 2023A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Trainingcited by 16
  11. 2023A Frustratingly Simple Decoding Method for Neural Text Generationcited by 16
  12. 2022Teacher Forcing Recovers Reward Functions for Text Generationcited by 11
7 more
  1. 2023Language Model Decoding as Direct Metrics Optimizationcited by 8
  2. 2022PreTR: Spatio-Temporal Non-Autoregressive Trajectory Prediction Transformercited by 6
  3. 2025Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learningcited by 5
  4. 2024LLMR: Knowledge Distillation with a Large Language Model-Induced Rewardcited by 3
  5. 2026Breaking the Likelihood Trap: Variance-Calibrated Modulation for Large Language Model Decodingcited by 1
  6. 2025CoVeR: Conformal Calibration for Versatile and Reliable Autoregressive Next-Token Predictioncited by 0
  7. 2026DharmaOCR: Specialized Small Language Models for Structured OCR that outperform Open-Source and Commercial Baselinescited by 0