# How Difficulty-Aware Staged Reinforcement Learning Enhances LLMs' Reasoning Capabilities: A Preliminary Experimental Study

arXiv: https://arxiv.org/abs/2504.00829 · HTML: https://mlsanity.com/paper/2504.00829/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 13 of 13)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [Process Reinforcement through Implicit Rewards](https://mlsanity.com/paper/2502.01456/) (2025)
- [LIMO: Less is More for Reasoning](https://mlsanity.com/paper/2502.03387/) (2025)
- [1.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Training](https://mlsanity.com/paper/2503.19633/) (2025)
- [Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinking](https://mlsanity.com/paper/2503.19855/) (2025)

## Built on by (most cited first, 14 of 14)

- [The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations](https://mlsanity.com/paper/2509.12886/) (2025)
- [Difficulty-Aware Agentic Orchestration for Query-Specific Multi-Agent Workflows](https://mlsanity.com/paper/2509.11079/) (2025)
- [DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Training](https://mlsanity.com/paper/2504.17565/) (2025)
- [Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Related Group Coding](https://mlsanity.com/paper/2505.21908/) (2025)
- [DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimization](https://mlsanity.com/paper/2508.14460/) (2025)
- [More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoning](https://mlsanity.com/paper/2510.07169/) (2025)
- [Verifying Meta-Awareness via Predictive Rewards in Reasoning Models](https://mlsanity.com/paper/2510.03259/) (2025)
- [LLMZero: Discovering Adaptive Training Strategies for RL Post-Training via LLM Agents](https://mlsanity.com/paper/2606.18388/) (2026)
- [Shorter but not Worse: Frugal Reasoning via Easy Samples as Length Regularizers in Math RLVR](https://mlsanity.com/paper/2511.01937/) (2025)
- [TACLer: Tailored Curriculum Reinforcement Learning for Efficient Reasoning](https://mlsanity.com/paper/2601.21711/) (2026)
- [FEVO: Financial Knowledge Expansion and Reasoning Evolution for Large Language Models](https://mlsanity.com/paper/2507.06057/) (2025)
- [Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals](https://mlsanity.com/paper/2605.22703/) (2026)
- [Adaptive Loss Balancing for Noise-Robust GRPO in Generative Recommendation](https://mlsanity.com/paper/2606.08480/) (2026)
- [Dynamic Important Example Mining for Reinforcement Finetuning](https://mlsanity.com/paper/2608.29252/) (2026)

## Abstract (arXiv)

Enhancing the reasoning capabilities of Large Language Models (LLMs) with efficiency and scalability remains a fundamental challenge in artificial intelligence research. This paper presents a rigorous experimental investigation into how difficulty-aware staged reinforcement learning (RL) strategies can substantially improve LLM reasoning performance. Through systematic analysis, we demonstrate that strategically selecting training data according to well-defined difficulty levels markedly enhances RL optimization. Moreover, we introduce a staged training methodology, progressively exposing models to increasingly challenging tasks, further amplifying reasoning capabilities. Our findings reveal significant cross-domain benefits when simultaneously training models on mathematical reasoning and code generation tasks. Notably, our proposed approach enables a 1.5B parameter model to achieve an accuracy of 42.3% on the AIME-2024 benchmark, 89.5% on the MATH-500 benchmark. These results underscore the efficacy of our method in advancing the reasoning proficiency of LLMs. We will open-source our datasets on GitHub and Hugging Face.