mlsanityData to 13 Oct 2026

Understanding the Repeat Curse in Large Language Models from a Feature Perspective

arXiv 2504.14218 ↗First posted 19 Apr 2025cs.CLcited by 37

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Large language models (LLMs) have made remarkable progress in various domains, yet they often suffer from repetitive text generation, a phenomenon we refer to as the "Repeat Curse". While previous studies have proposed decoding strategies to mitigate repetition, the underlying mechanism behind this issue remains insufficiently explored. In this work, we investigate the root causes of repetition in LLMs through the lens of mechanistic interpretability. Inspired by recent advances in Sparse Autoencoders (SAEs), which enable monosemantic feature extraction, we propose a novel approach, "Duplicatus Charm", to induce and analyze the Repeat Curse. Our method systematically identifies "Repetition Features" -the key model activations responsible for generating repetitive outputs. First, we locate the layers most involved in repetition through logit analysis. Next, we extract and stimulate relevant features using SAE-based activation manipulation. To validate our approach, we construct a repetition dataset covering token and paragraph level repetitions and introduce an evaluation pipeline to quantify the influence of identified repetition features. Furthermore, by deactivating these features, we have effectively mitigated the Repeat Curse. The source code of our work is publicly available at: https://github.com/kaustpradalab/repeat-curse-llm

Key facts. Cited by 37 papers in the map (data through 13 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 32 of its 55 references that are in the map. First posted on arXiv 19 Apr 2025.

Builds on

32 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders27 Oct
  2. 2024Applying sparse autoencoders to unlearn knowledge in language models25 Oct
  3. 2024Improving Dictionary Learning with Gated Sparse Autoencoders24 Apr
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 28

  1. 2024The Llama 3 Herd of Modelscited by 14,486
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,541
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,893
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,323
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,145
  6. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 420
  7. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 393
  8. 2022A Contrastive Framework for Neural Text Generationcited by 224
  9. 2023DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Textcited by 207
  10. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  11. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 179
  12. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 107
16 more
  1. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 91
  2. 2020A Theoretical Analysis of the Repetition Problem in Text Generationcited by 83
  3. 2024InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoderscited by 66
  4. 2023Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspectivecited by 39
  5. 2024Locate-then-edit for Multi-hop Factual Recall under Knowledge Editingcited by 36
  6. 2025Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducementscited by 32
  7. 2024PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluationcited by 26
  8. 2024Repetition Neurons: How Do Language Models Produce Repetitions?cited by 23
  9. 2025EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identificationcited by 21
  10. 2024Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editingcited by 18
  11. 2024Learning biologically relevant features in a pathology foundation model using sparse autoencoderscited by 14
  12. 2023Humans and language models diverge when predicting repeating textcited by 13
  13. 2023Penalty Decoding: Well Suppress the Self-Reinforcement Effect in Open-Ended Text Generationcited by 13
  14. 2025Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoningcited by 11
  15. 2024Understanding Reasoning in Chain-of-Thought from the Hopfieldian Viewcited by 10
  16. 2024The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generationcited by 9

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 16; 2026: 21.

37 papers.

Sort:

By area:

  1. 2025When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Modelscited by 49
  2. 2025Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Modelscited by 35
  3. 2025AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestrationcited by 26
  4. 2025Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Modelscited by 16
  5. 2025PIXEL: Adaptive Steering Via Position-wise Injection with eXact Estimated Levels under Subspace Calibrationcited by 10
  6. 2025COMPKE: Complex Question Answering under Knowledge Editingcited by 6
  7. 2025Understanding and Mitigating Political Stance Cross-topic Generalization in Large Language Modelscited by 6
  8. 2026Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspectivecited by 6
  9. 2025What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformerscited by 5
  10. 2025DTS: Enhancing Large Reasoning Models via Decoding Tree Sketchingcited by 5
  11. 2026Controlling Repetition in Protein Language Modelscited by 5
  12. 2025Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localizationcited by 4
25 more
  1. 2025The Compositional Architecture of Regret in Large Language Modelscited by 3
  2. 2025Stop Before You Fail: Operational Capability Boundaries for Mitigating Unproductive Reasoning in Large Reasoning Modelscited by 3
  3. 2026Concept-Based Dictionary Learning for Inference-Time Safety in Vision Language Action Modelscited by 3
  4. 2026MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphscited by 3
  5. 2026Echoes as Anchors: Probabilistic Costs and Attention Refocusing in LLM Reasoningcited by 2
  6. 2026Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Imagescited by 2
  7. 2026Word Recovery in Large Language Models Enables Character-Level Tokenization Robustnesscited by 2
  8. 2025POLIS-Bench: Towards Multi-Dimensional Evaluation of LLMs for Bilingual Policy Tasks in Governmental Scenarioscited by 1
  9. 2026Faithful-Patchscopes: Understanding and Mitigating Model Bias in Hidden Representations Explanation of Large Language Modelscited by 1
  10. 2026VideoSTF: Stress-Testing Output Repetition in Video Large Language Modelscited by 1
  11. 2026SinhaLegal: A Benchmark Corpus for Information Extraction and Analysis in Sinhala Legislative Textscited by 1
  12. 2026VecCISC: Improving Confidence-Informed Self-Consistency with Reasoning Trace Clustering and Candidate Answer Selectioncited by 1
  13. 2026Sanity Checks for Long-Form Hallucination Detectioncited by 1
  14. 2026Breaking the Likelihood Trap: Variance-Calibrated Modulation for Large Language Model Decodingcited by 1
  15. 2026Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learningcited by 1
  16. 2025Evaluating Large Language Models for Functional and Maintainable Code in Industrial Settings: A Case Study at ASMLcited by 0
  17. 2025In-Context Learning Without Copyingcited by 0
  18. 2025Unsupervised Memorability Modeling from Tip-of-the-Tongue Retrieval Queriescited by 0
  19. 2026Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencoderscited by 0
  20. 2026DharmaOCR: Specialized Small Language Models for Structured OCR that outperform Open-Source and Commercial Baselinescited by 0
  21. 2026From Representations to Behaviors: Exploring the Person-Situation-Behavior Triad in LLMscited by 0
  22. 2026CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversitycited by 0
  23. 2026Don't Repeat Yourself: Stopping Verbatim Loops at Sampling Timecited by 0
  24. 2026Uncovering Uncontrolled Repetition through Residual Stream Dynamicscited by 0
  25. 2026Not All Answers Are Contextually Persuadable: Inference Dynamics in Large Language Models under Contextual Influencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Scaling Laws and Interpretability of Learning from Repeated Data
  2. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  3. 2025Interpreting and Steering LLMs with Mutual Information-based Explanations on Sparse Autoencoders
  4. 2025Rethinking Repetition Problems of LLMs in Code Generation
  5. 2025Antislop: A Comprehensive Framework for Identifying and Eliminating Repetitive Patterns in Language Models
  6. 2026PARTREP: Learning What to Repeat for Decoder-only LLMs
  7. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
  8. 2024When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models