mlsanityData to 7 Oct 2026

Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning

arXiv 2504.03380 ↗First posted 4 Apr 2025cs.CL cs.AIcited by 100

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent advances in reinforcement learning with verifiable rewards (RLVR) show that large language models enhance their reasoning abilities when trained with verifiable signals. However, due to reward sparsity, effectiveness depends heavily on selecting samples of appropriate difficulty. In this work, we present a formal analysis of online difficulty-aware filtering and establish its theoretical foundations. We show that expected policy improvement is lower-bounded by the variance of task-level success probabilities, implying that selecting tasks of intermediate difficulty maximizes learning efficiency. Building on this, we demonstrate that balanced filtering maximizes this lower bound, leading to superior performance and sample efficiency. Evaluations across multiple math reasoning benchmarks validate that balanced filtering consistently enhances convergence speed and final performance, achieving up to +12% gains in less than half the training steps of standard GRPO. By extending our analysis to various reward distributions, we provide a principled foundation for future RLVR curriculum strategies, confirmed through both theoretical analysis and extensive empirical results.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 68 references that are in the map. First posted on arXiv 4 Apr 2025.

Builds on

38 of 68 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification9 Mar
  2. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution25 Feb
  3. 2025LIMR: Less is More for RL Scaling17 Feb
  4. 2025Process Reinforcement through Implicit Rewards3 Feb
  5. 2024Teaching Large Language Models to Reason with Reinforcement Learning7 Mar
  6. 2023Let's Verify Step by Step31 May
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2025s1: Simple test-time scalingcited by 1,311
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  11. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
18 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  2. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  3. 2025LIMO: Less is More for Reasoningcited by 488
  4. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  5. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  6. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  7. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  8. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 201
  9. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  10. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  11. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  12. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  13. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 49
  14. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  15. 2021The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Modelscited by 36
  16. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 34
  17. 2025What is the Alignment Objective of GRPO?cited by 23
  18. 2024Curriculum Learning for Small Code Language Modelscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 43; 2026: 57.

100 papers.

Sort:

By area:

  1. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  2. 2025Self-Evolving Curriculum for LLM Reasoningcited by 107
  3. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 95
  4. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoningcited by 89
  5. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 88
  6. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 68
  7. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 48
  8. 2025DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-trainingcited by 45
  9. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 33
  10. 2025Diversity-Incentivized Exploration for Versatile Reasoningcited by 28
  11. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learningcited by 27
  12. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Modelscited by 27
38 more
  1. 2025VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoningcited by 26
  2. 2025Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulationcited by 20
  3. 2025How to Train Your Advisor: Steering Black-Box LLMs with Advisor Modelscited by 20
  4. 2025TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoningcited by 18
  5. 2025HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocationcited by 18
  6. 2025CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMscited by 16
  7. 2025Play to Generalize: Learning to Reason Through Game Playcited by 15
  8. 2025RL for Reasoning by Adaptively Revealing Rationalescited by 15
  9. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  10. 2026Teaching Models to Teach Themselves: Reasoning at the Edge of Learnabilitycited by 13
  11. 2026Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillationcited by 13
  12. 2025Do Thinking Tokens Help or Trap? Towards More Efficient Large Reasoning Modelcited by 12
  13. 2025BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuningcited by 12
  14. 2025Data-Efficient RLVR via Off-Policy Influence Guidancecited by 12
  15. 2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learningcited by 11
  16. 2025InfoFlow: Reinforcing Search Agent Via Reward Density Optimizationcited by 11
  17. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 11
  18. 2025G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidancecited by 10
  19. 2025Writing-RL: Advancing Long-form Writing via Adaptive Curriculum Reinforcement Learningcited by 9
  20. 2026Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Textcited by 9
  21. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 9
  22. 2026Understanding Diversity Collapse in RLVR via the Lens of Overtrainingcited by 9
  23. 2025Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment Perspectivecited by 8
  24. 2025LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignmentcited by 8
  25. 2026Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Modelscited by 8
  26. 2026Prune as You Generate: Online Rollout Pruning for Faster and Better RLVRcited by 8
  27. 2025Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Related Group Codingcited by 7
  28. 2025EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learningcited by 7
  29. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Explorationcited by 7
  30. 2026Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptationcited by 7
  31. 2026Prompt replay: speeding up grpo with on-policy reuse of high-signal promptscited by 7
  32. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoningcited by 7
  33. 2025The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedbackcited by 6
  34. 2025HyperCLOVA X THINK Technical Reportcited by 5
  35. 2026From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoningcited by 5
  36. 2025Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Trainingcited by 4
  37. 2026When Sharpening Becomes Collapse: Sampling Bias and Semantic Coupling in RL with Verifiable Rewardscited by 4
  38. 2026Efficient RLVR Training via Weighted Mutual Information Data Selectioncited by 4

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  2. 2026Efficient RLVR Scheduling via Graph-Structured Online Difficulty Estimation
  3. 2025Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward
  4. 2025From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning
  5. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  6. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
  7. 2026Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learning
  8. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement