mlsanityData to 14 Oct 2026

Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment Perspective

arXiv 2505.17652 ↗First posted 23 May 2025cs.LG cs.AIcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning exhibits potential in enhancing the reasoning abilities of large language models, yet it is hard to scale for the low sample efficiency during the rollout phase. Existing methods attempt to improve efficiency by scheduling problems based on problem difficulties. However, these approaches suffer from unstable and biased estimations of problem difficulty and fail to capture the alignment between model competence and problem difficulty in RL training, leading to suboptimal results. To tackle these limitations, this paper introduces Competence-Difficulty Alignment Sampling (CDAS), which enables accurate and stable estimation of problem difficulties by aggregating historical performance discrepancies of problems. Then the model competence is quantified to adaptively select problems whose difficulty is in alignment with the model's current competence using a fixed-point system. Experimental results across a range of challenging mathematical benchmarks show that CDAS achieves great improvements in both accuracy and efficiency. CDAS attains the highest average accuracy against baselines and exhibits significant speed advantages compared to Dynamic Sampling, a competitive strategy in DAPO, which is 2.33 times slower than CDAS.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 41 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

25 of 41 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  2. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  4. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  7. 2025LIMR: Less is More for RL Scaling17 Feb
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  10. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 14

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  6. 2024OpenAI o1 System Cardcited by 1,244
  7. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2021Measuring Coding Challenge Competence With APPScited by 902
  10. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  11. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  12. 2023TACO: Topics in Algorithmic COde generation datasetcited by 160
2 more
  1. 2024Planning In Natural Language Improves LLM Search For Code Generationcited by 83
  2. 2023Stabilizing RLHF through Advantage Model and Selective Rehearsalcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 17
  2. 2026Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learningcited by 14
  3. 2026Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Modelscited by 4
  4. 2025Diffuse Thinking: Exploring Diffusion Language Models as Efficient Thought Proposers for Reasoningcited by 3
  5. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinementcited by 3
  6. 2026ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillationcited by 3
  7. 2025Counteracting Matthew Effect in Self-Improvement of LVLMs through Head-Tail Re-balancingcited by 0
  8. 2026CATPO: Critique-Augmented Tree Policy Optimizationcited by 0
  9. 2026DART-ES: Difficulty-Aware Reweighting and Targeted Replay for Fine-Tuning LLMs with Evolution Strategiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025How Difficulty-Aware Staged Reinforcement Learning Enhances LLMs' Reasoning Capabilities: A Preliminary Experimental Study
  2. 2025From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization
  3. 2025Learning to Reason at the Frontier of Learnability
  4. 2025Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation
  5. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration
  6. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning
  7. 2026Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data
  8. 2024Teaching Large Language Models to Reason with Reinforcement Learning