mlsanityData to 14 Oct 2026

Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective

arXiv 2310.10226 ↗First posted 16 Oct 2023cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

There are a number of diverging hypotheses about the neural text degeneration problem, i.e., generating repetitive and dull loops, which makes this problem both interesting and confusing. In this work, we aim to advance our understanding by presenting a straightforward and fundamental explanation from the data perspective. Our preliminary investigation reveals a strong correlation between the degeneration issue and the presence of repetitions in training data. Subsequent experiments also demonstrate that by selectively dropping out the attention to repetitive words in training data, degeneration can be significantly minimized. Furthermore, our empirical analysis illustrates that prior works addressing the degeneration issue from various standpoints, such as the high-inflow words, the likelihood objective, and the self-reinforcement phenomenon, can be interpreted by one simple explanation. That is, penalizing the repetitions in training data is a common and fundamental factor for their effectiveness. Moreover, our experiments reveal that penalizing the repetitions in training data remains critical even when considering larger model sizes and instruction tuning.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

31 of 43 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  12. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
19 more
  1. 2018Hierarchical Neural Story Generationcited by 1,193
  2. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  3. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  4. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  5. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  6. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  7. 2022A Contrastive Framework for Neural Text Generationcited by 225
  8. 2022Locally Typical Samplingcited by 111
  9. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 94
  10. 2020A Theoretical Analysis of the Repetition Problem in Text Generationcited by 83
  11. 2022Contrastive Search Is What You Need For Neural Text Generationcited by 59
  12. 2021Non-Autoregressive Text Generation with Pre-trained Language Modelscited by 36
  13. 2021Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generationcited by 19
  14. 2021Relating Neural Text Degeneration to Exposure Biascited by 19
  15. 2022A Simple Contrastive Learning Objective for Alleviating Neural Text Degenerationcited by 18
  16. 2023A Frustratingly Simple Decoding Method for Neural Text Generationcited by 16
  17. 2022An Empirical Study On Contrastive Search And Contrastive Decoding For Open-ended Text Generationcited by 14
  18. 2022A Continuum of Generation Tasks for Investigating Length Bias and Degenerate Repetitioncited by 3
  19. 2022Generating Repetitions with Appropriate Repeated Wordscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 14; 2026: 19.

39 papers.

Sort:

By area:

  1. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 114
  2. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  3. 2025RLVR-World: Training World Models with Reinforcement Learningcited by 40
  4. 2025Understanding the Repeat Curse in Large Language Models from a Feature Perspectivecited by 37
  5. 2025AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestrationcited by 27
  6. 2024Repetition Neurons: How Do Language Models Produce Repetitions?cited by 23
  7. 2025ThinkTrap: Denial-of-Service Attacks against Black-box LLM Services via Infinite Thinkingcited by 23
  8. 2024Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Modelscited by 15
  9. 2024ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coveragecited by 12
  10. 2025Repetitions are not all alike: distinct mechanisms sustain repetition in language modelscited by 10
  11. 2024Pap2Pat: Benchmarking Outline-Guided Long-Text Patent Generation with Patent-Paper Pairscited by 8
  12. 2025What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformerscited by 7
27 more
  1. 2026Lookahead-then-Verify: Reliable Constrained Decoding for Diffusion LLMs under Context-Free Grammarscited by 7
  2. 2026Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspectivecited by 6
  3. 2026Enhancing Instruction Following of LLMs via Activation Steering with Dynamic Rejectioncited by 6
  4. 2025USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agentscited by 5
  5. 2025Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learningcited by 5
  6. 2025EfficientUICoder: A Bidirectional Token Compression Framework for Efficient MLLM-Based UI Code Generationcited by 5
  7. 2025Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generationcited by 4
  8. 2026OmniGen-AR: AutoRegressive Any-to-Image Generationcited by 4
  9. 2025Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Modelscited by 3
  10. 2025Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Riskscited by 2
  11. 2026Resource Consumption Threats in Large Language Modelscited by 2
  12. 2026LoopGuard: Breaking Self-Reinforcing Attention Loops via Dynamic KV Cache Interventioncited by 2
  13. 2026VecCISC: Improving Confidence-Informed Self-Consistency with Reasoning Trace Clustering and Candidate Answer Selectioncited by 2
  14. 2025Leveraging Importance Sampling to Detach Alignment Modules from Large Language Modelscited by 1
  15. 2025Score Before You Speak: Improving Persona Consistency in Dialogue Generation using Response Quality Scorescited by 1
  16. 2026TeleDoCTR: Domain-Specific and Contextual Troubleshooting for Telecommunicationscited by 1
  17. 2026VideoSTF: Stress-Testing Output Repetition in Video Large Language Modelscited by 1
  18. 2026Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Codecited by 1
  19. 2026Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learningcited by 1
  20. 2026CUSUM-Shaped Inference-Time Monitoring and Targeted Re-Decoding for Quantized Small Language Model Reasoningcited by 1
  21. 2026DharmaOCR: Specialized Small Language Models for Structured OCR that outperform Open-Source and Commercial Baselinescited by 0
  22. 2026Escaping Mode Collapse in LLM Generation via Geometric Regulationcited by 0
  23. 2026SOMA: Efficient Multi-turn LLM Serving via Small Language Modelcited by 0
  24. 2026Calibrated e-CUSUM Decoding for Quantized Reasoning Models: Why Token Log-Probability Is the Wrong Observable for Decoding Monitorscited by 0
  25. 2026Patterns of Priming in Production: Lexical, Semantic and Structural Alignment in Language Model Generationcited by 0
  26. 2026Target-Language Generation in Multilingual Models: Activation Steering and Optimal Controlcited by 0
  27. 2026What a Cross-Model Fixed-Point Census Can and Cannot Arbitrate About Repetitioncited by 0