mlsanityData to 7 Oct 2026

Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability

arXiv 2601.18778 ↗First posted 26 Jan 2026cs.LG cs.CLcited by 13

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

RL methods for scaling large reasoning models stall on datasets with low initial success rates, and thus little training signal. We investigate a fundamental question: Can a pretrained LLM leverage latent knowledge to generate an automated curriculum for problems it cannot solve? We explore this with SOAR: An asymmetric self-play framework that uses meta-RL to surface these pedagogical signals. A teacher model proposes synthetic problems for a student model, and is rewarded with its improvement on a subset of hard problems, thus grounding the curriculum in real student progress rather than intrinsic proxy rewards. Our study on the hardest subsets of math benchmarks (0/128 success) reveals three core findings. First, it is possible to realize bilevel meta-RL that unlocks learning under sparse, binary rewards by sharpening a latent capacity of pretrained models to generate useful problems. Second, grounded rewards outperform intrinsic learnability rewards used in prior LLM self-play, reliably avoiding typical instability and diversity collapse modes. Third, the structure and well-posedness of questions are more critical for learning progress than solution correctness. Our results suggest that the ability to generate useful stepping stones does not require the preexisting ability to solve the hard problems, paving a principled path to escape reasoning plateaus without additional curated data

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 55 of its 68 references that are in the map. First posted on arXiv 26 Jan 2026.

Builds on

55 of 68 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Towards Understanding Self-play for LLM Reasoning31 Oct
  2. 2025Outcome-based Exploration for LLM Reasoning8 Sep
  3. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning7 Jun
  4. 2025Maximizing Confidence Alone Improves Reasoning28 May
  5. 2025Can Large Reasoning Models Self-Train?27 May
  6. 2025Learning to Reason without External Rewards26 May
  7. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data25 May
  8. 2025Self-Evolving Curriculum for LLM Reasoning20 May
  9. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  10. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  11. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  12. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
4 more
  1. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  4. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan

Other references · 39

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  5. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  6. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  7. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  8. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  9. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  10. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  11. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  12. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 299
27 more
  1. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  2. 2017Automated Curriculum Learning for Neural Networkscited by 251
  3. 2018Dataset Distillationcited by 224
  4. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  5. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  6. 2020Prioritized Level Replaycited by 190
  7. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  8. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  9. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  10. 2021Replay-Guided Adversarial Environment Designcited by 108
  11. 2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
  12. 2020Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Surveycited by 83
  13. 2018CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learningcited by 78
  14. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  15. 2025Self-Adapting Language Modelscited by 71
  16. 2025Self-Questioning Language Modelscited by 47
  17. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  18. 2025Language Self-Play For Data-Free Trainingcited by 42
  19. 2017Early Stopping without a Validation Setcited by 39
  20. 2024HARP: A challenging human-annotated math reasoning benchmarkcited by 25
  21. 2025RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?cited by 23
  22. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  23. 2025Improving the Scaling Laws of Synthetic Data with Deliberate Practicecited by 17
  24. 2026Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMscited by 9
  25. 2025How Reinforcement Learning After Next-Token Prediction Facilitates Learningcited by 8
  26. 2023Regularity as Intrinsic Reward for Free Playcited by 5
  27. 2025Revisiting Generalization Across Difficulty Levels: It's Not So Easycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026Scaling Self-Play with Self-Guidancecited by 9
  2. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 6
  3. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  4. 2026GASP: Guided Asymmetric Self-Play For Coding LLMscited by 4
  5. 2026Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problemscited by 4
  6. 2026AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signalscited by 3
  7. 2026Policy Improvement Reinforcement Learningcited by 2
  8. 2026PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Playcited by 2
  9. 2026Question Begets Question: Self-Evolving Curriculum for Reinforcement Fine-Tuning on Competition Mathematicscited by 2
  10. 2026ANCORA: Learning to Question via Manifold-Anchored Self-Play for Verifiable Reasoningcited by 0
  11. 2026CompassPlay: Rewarding the Proposer for Where It Moves the Solvercited by 0
  12. 2026Frontier Learning: Training LLM Reasoners at the Edge of Capabilitycited by 0
1 more
  1. 2026A Good Self-Teacher Meets the Student Where They Are: Joint On-Policy Learning and Teachingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Unlocking the Unsolvable: Teacher-Guided Curriculum for Data-Efficient RLVR
  2. 2025Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation
  3. 2026Ladders of Thought: A Self-Evolving Curriculum of Progressively Simplified Reasoning Traces
  4. 2025Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning
  5. 2026DARC: Decoupled Asymmetric Reasoning Curriculum for LLM Evolution
  6. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning
  7. 2026Preventing Curriculum Collapse in Self-Evolving Reasoning Systems
  8. 2026R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training