mlsanityData to 14 Oct 2026

Learning to Reason at the Frontier of Learnability

arXiv 2502.12272 ↗First posted 17 Feb 2025cs.LG cs.AI cs.CLcited by 23

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning is now widely adopted as the final stage of large language model training, especially for reasoning-style tasks such as maths problems. Typically, models attempt each question many times during a single training step and attempt to learn from their successes and failures. However, we demonstrate that throughout training with two popular algorithms (PPO and VinePPO) on two widely used datasets, many questions are either solved by all attempts - meaning they are already learned - or by none - providing no meaningful training signal. To address this, we adapt a method from the reinforcement learning literature - sampling for learnability - and apply it to the reinforcement learning stage of LLM training. Our curriculum prioritises questions with high variance of success, i.e. those where the agent sometimes succeeds, but not always. Our findings demonstrate that this curriculum consistently boosts training performance across multiple algorithms and datasets, paving the way for more efficient and effective reinforcement learning with LLMs.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 45 of its 63 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

45 of 63 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  2. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  4. 2024Group Robust Preference Optimization in Reward-free RLHF30 May
  5. 2024Teaching Large Language Models to Reason with Reinforcement Learning7 Mar
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 39

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2016OpenAI Gymcited by 1,736
  4. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  7. 2024Measuring short-form factuality in large language modelscited by 373
  8. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  9. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  10. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  11. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
  12. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
27 more
  1. 2017Automated Curriculum Learning for Neural Networkscited by 257
  2. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  3. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  4. 2017Teacher-Student Curriculum Learningcited by 210
  5. 2020Prioritized Level Replaycited by 198
  6. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  7. 2022Evolving Curricula with Regret-Based Environment Designcited by 173
  8. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  9. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  10. 2015Online Batch Selection for Faster Training of Neural Networkscited by 122
  11. 2018Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networkscited by 114
  12. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 102
  13. 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
  14. 2020Automatic Curriculum Learning through Value Disagreementcited by 87
  15. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 80
  16. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  17. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 63
  18. 2020Goal-GAN: Multimodal Trajectory Prediction Based on Goal Position Estimationcited by 57
  19. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 55
  20. 2019Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimizationcited by 51
  21. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 50
  22. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasingcited by 39
  23. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 31
  24. 2023Proximal Curriculum for Reinforcement Learning Agentscited by 20
  25. 2019CLIC: Curriculum Learning and Imitation for object Control in non-rewarding environmentscited by 10
  26. 2020Information-theoretic Task Selection for Meta-Reinforcement Learningcited by 10
  27. 2024Towards A Comprehensive Assessment of AI's Environmental Impactcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 14.

23 papers.

Sort:

By area:

  1. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoningcited by 92
  2. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Modelscited by 75
  3. 2025MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resourcescited by 39
  4. 2025Training a Generally Curious Agentcited by 31
  5. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 31
  6. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learningcited by 29
  7. 2025VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoningcited by 26
  8. 2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 14
  9. 2025Bootstrapping Task Spaces for Self-Improvementcited by 10
  10. 2026Prompt replay: speeding up grpo with on-policy reuse of high-signal promptscited by 8
  11. 2026Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problemscited by 4
  12. 2025Unsupervised Partner Design Enables Robust Ad-hoc Teamworkcited by 3
11 more
  1. 2026Rotation-Preserving Supervised Fine-Tuningcited by 3
  2. 2026Rate or Fate? RLV^varepsilonR: Reinforcement Learning with Verifiable Noisy Rewardscited by 1
  3. 2026Improving Regret Approximation for Unsupervised Dynamic Environment Generationcited by 1
  4. 2026Beyond Normalization: Rethinking the Partition Function as a Difficulty Scheduler for RLVRcited by 1
  5. 2026D'ej`aQ: Open-Ended Evolution of Diverse, Learnable and Verifiable Problemscited by 0
  6. 2026Manifold Bandits: Bayesian Curriculum Learning over the Latent Geometry of Large Language Modelscited by 0
  7. 2026AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop?cited by 0
  8. 2026Frontier Learning: Training LLM Reasoners at the Edge of Capabilitycited by 0
  9. 2026Prompts Live on an Arc: Gaussian Curricula in Fisher--Rao Coordinates for Rollout-Efficient GRPOcited by 0
  10. 2026Understanding Enrichment in Reinforcement Learningcited by 0
  11. 2026DART-ES: Difficulty-Aware Reweighting and Targeted Replay for Fine-Tuning LLMs with Evolution Strategiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025How Difficulty-Aware Staged Reinforcement Learning Enhances LLMs' Reasoning Capabilities: A Preliminary Experimental Study
  2. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation
  3. 2025CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs
  4. 2025Self-Evolving Curriculum for LLM Reasoning
  5. 2025Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation
  6. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
  7. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
  8. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions