mlsanityData to 14 Oct 2026

Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges

arXiv 2502.01612 ↗First posted 3 Feb 2025cs.LG cs.AIcited by 31

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Large language models often struggle with length generalization and solving complex problem instances beyond their training distribution. We present a self-improvement approach where models iteratively generate and learn from their own solutions, progressively tackling harder problems while maintaining a standard transformer architecture. Across diverse tasks including arithmetic, string manipulation, and maze solving, self-improving enables models to solve problems far beyond their initial training distribution-for instance, generalizing from 10-digit to 100-digit addition without apparent saturation. We observe that in some cases filtering for correct self-generated examples leads to exponential improvements in out-of-distribution performance across training rounds. Additionally, starting from pretrained models significantly accelerates this self-improvement process for several tasks. Our results demonstrate how controlled weak-to-strong curricula can systematically teach a model logical extrapolation without any changes to the positional embeddings, or the model architecture.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 60 of its 67 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

60 of 67 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Self-Improvement in Language Models: The Sharpening Mechanism2 Dec
  2. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improve25 Jul
  3. 2024Self-Rewarding Language Models18 Jan
  4. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug

Other references · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  7. 2023Teaching Large Language Models to Self-Debugcited by 988
  8. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  9. 2022Large Language Models Can Self-Improvecited by 664
  10. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  11. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  12. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
38 more
  1. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  2. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  3. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  4. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  5. 2023Self-Consuming Generative Models Go MADcited by 250
  6. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Stepcited by 235
  7. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  8. 2022Exploring Length Generalization in Large Language Modelscited by 206
  9. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  10. 2017Deep Learning is Robust to Massive Label Noisecited by 165
  11. 2024The pitfalls of next-token predictioncited by 160
  12. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 151
  13. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 130
  14. 2020From Local Structures to Size Generalization in Graph Neural Networkscited by 124
  15. 2023Teaching Arithmetic to Small Transformerscited by 123
  16. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 117
  17. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 117
  18. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 100
  19. 2024Looped Transformers for Length Generalizationcited by 99
  20. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 96
  21. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 86
  22. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 81
  23. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 79
  24. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 78
  25. 2023Length Generalization in Arithmetic Transformerscited by 66
  26. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 61
  27. 2022Will Large-scale Generative Models Corrupt Future Datasets?cited by 59
  28. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  29. 2020The EOS Decision and Length Extrapolationcited by 46
  30. 2023Combining Generative Artificial Intelligence (AI) and the Internet: Heading towards Evolution or Degradation?cited by 41
  31. 2024The Unreasonable Effectiveness of Easy Training Data for Hard Taskscited by 40
  32. 2024Self-Correcting Self-Consuming Loops for Generative Model Trainingcited by 37
  33. 2023Understanding Addition in Transformerscited by 34
  34. 2024PatternBoost: Constructions in Mathematics with a Little Help from AIcited by 34
  35. 2019Location Attention for Extrapolation to Longer Sequencescited by 33
  36. 2024I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigmcited by 27
  37. 2024Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structurecited by 23
  38. 2024Weak-to-Strong Generalization Through the Data-Centric Lenscited by 21

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 11.

31 papers.

Sort:

By area:

  1. 2025Can Large Reasoning Models Self-Train?cited by 68
  2. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 64
  3. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 33
  4. 2025RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learningcited by 30
  5. 2026Large Language Model Reasoning Failurescited by 21
  6. 2025Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalizationcited by 14
  7. 2025Extrapolation by Association: Length Generalization Transfer in Transformerscited by 13
  8. 2025Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learningcited by 9
  9. 2025h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learningcited by 9
  10. 2025The Role of Sparsity for Length Generalization in Transformerscited by 8
  11. 2025Quantitative Bounds for Length Generalization in Transformerscited by 8
  12. 2025Self-Supervised Transformers as Iterative Solution Improvers for Constraint Satisfactioncited by 6
19 more
  1. 2025How Does RL Post-training Induce Skill Composition? A Case Study on Countdowncited by 6
  2. 2025Symmetry-Aware Transformer Training for Automated Planningcited by 5
  3. 2025Revisiting Generalization Across Difficulty Levels: It's Not So Easycited by 5
  4. 2026Self-Improving World Modelling with Latent Actionscited by 5
  5. 2025Reasoning Models Reason Well, Until They Don'tcited by 4
  6. 2025Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Trainingcited by 4
  7. 2026Learning to Reason with Curriculum I: Provable Benefits of Autocurriculumcited by 4
  8. 2026Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stoppingcited by 4
  9. 2025Maximizing Prefix-Confidence at Test-Time Efficiently Improves Mathematical Reasoningcited by 3
  10. 2025Provably Learning from Modern Language Models via Low Logit Rankcited by 3
  11. 2026Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmeticcited by 3
  12. 2026On the Role of Computation in Reinforcement Learningcited by 3
  13. 2025Knot So Simple: A Minimalistic Environment for Spatial Reasoningcited by 2
  14. 2026On the Ability of Transformers to Verify Planscited by 2
  15. 2026Transformers in the Dark: Navigating Unknown Search Spaces via Bandit Feedbackcited by 2
  16. 2026Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architecturescited by 1
  17. 2026From Shortcut Learning to Discrete Neural Insertion Sortcited by 1
  18. 2025Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolationcited by 0
  19. 2026The Value of Being Wrong: Self-Mined Visual In-Context Learning from Errorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Teaching Language Models to Self-Improve through Interactive Demonstrations
  2. 2024Can Large Language Models Invent Algorithms to Improve Themselves?: Algorithm Discovery for Recursive Self-Improvement through Reinforcement Learning
  3. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
  4. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
  5. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  6. 2026A Task-Centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula
  7. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gap
  8. 2025The Self-Improvement Paradox: Can Language Models Bootstrap Reasoning Capabilities without External Scaffolding?