# Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation arXiv: https://arxiv.org/abs/2601.20614 ยท HTML: https://mlsanity.com/paper/2601.20614/ ## Key facts - Cited by 19 papers in the map (data through 14 Oct 2026). - Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. - Builds on 34 of its 42 references that are in the map. - First posted on arXiv 28 Jan 2026. ## Placed in - LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/) ## Builds on (most cited first, 30 of 34) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025) - [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025) - [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021) - [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025) - [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025) - [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025) - [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024) - [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024) - [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024) - [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025) - [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023) - [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025) - [Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs](https://mlsanity.com/paper/2506.14245/) (2025) - [VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks](https://mlsanity.com/paper/2504.05118/) (2025) - [GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning](https://mlsanity.com/paper/2105.14517/) (2021) - [GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning](https://mlsanity.com/paper/2504.02546/) (2025) - [Geometric-Mean Policy Optimization](https://mlsanity.com/paper/2507.20673/) (2025) - [Common 7B Language Models Already Possess Strong Math Capabilities](https://mlsanity.com/paper/2403.04706/) (2024) - [S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models](https://mlsanity.com/paper/2505.07686/) (2025) - [GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models](https://mlsanity.com/paper/2504.09696/) (2025) - [Tree Search for LLM Agent Reinforcement Learning](https://mlsanity.com/paper/2509.21240/) (2025) - [Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR](https://mlsanity.com/paper/2508.14029/) (2025) - [MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning](https://mlsanity.com/paper/2310.05506/) (2023) - [Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspective](https://mlsanity.com/paper/2505.12886/) (2025) - [GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning](https://mlsanity.com/paper/2507.10628/) (2025) - [AdaCuRL: Adaptive Curriculum Reinforcement Learning with Invalid Sample Mitigation and Historical Revisiting](https://mlsanity.com/paper/2511.09478/) (2025) - [HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocation](https://mlsanity.com/paper/2505.19866/) (2025) ## Built on by (most cited first, 19 of 19) - [Code2World: A GUI World Model via Renderable Code Generation](https://mlsanity.com/paper/2602.09856/) (2026) - [Visually-Guided Policy Optimization for Multimodal Reasoning](https://mlsanity.com/paper/2604.09349/) (2026) - [FASA: Frequency-aware Sparse Attention](https://mlsanity.com/paper/2602.03152/) (2026) - [MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios](https://mlsanity.com/paper/2602.22638/) (2026) - [Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation](https://mlsanity.com/paper/2602.05548/) (2026) - [Learning Agentic Policy from Action Guidance](https://mlsanity.com/paper/2605.12004/) (2026) - [Leveraging Error Diversity in Group Rollouts for Reinforcement Learning](https://mlsanity.com/paper/2605.17333/) (2026) - [Policy Improvement Reinforcement Learning](https://mlsanity.com/paper/2604.00860/) (2026) - [Ace-Skill: Bootstrapping Multimodal Agents with Prioritized and Clustered Evolution](https://mlsanity.com/paper/2605.08887/) (2026) - [Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoning](https://mlsanity.com/paper/2605.27765/) (2026) - [RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents](https://mlsanity.com/paper/2606.19047/) (2026) - [PrAg-PO: Prompt Augmented Policy Optimization for Robust and Diverse Mathematical Reasoning](https://mlsanity.com/paper/2602.03190/) (2026) - [expo: Exploration-prioritized policy optimization via adaptive kl regulation and gaussian curriculum sampling](https://mlsanity.com/paper/2605.09923/) (2026) - [fg-expo: Frontier-guided exploration-prioritized policy optimization via adaptive kl and gaussian curriculum](https://mlsanity.com/paper/2605.11403/) (2026) - [RISE: Reliable Improvement in Self-Evolving Vision-Language Models](https://mlsanity.com/paper/2605.20914/) (2026) - [Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVR](https://mlsanity.com/paper/2609.40115/) (2026) - [D2Evo: Dual Difficulty-Aware Self-Evolution for Data-Efficient Reinforcement Learning](https://mlsanity.com/paper/2605.17037/) (2026) - [Frontier Learning: Training LLM Reasoners at the Edge of Capability](https://mlsanity.com/paper/2609.35426/) (2026) - [DART-ES: Difficulty-Aware Reweighting and Targeted Replay for Fine-Tuning LLMs with Evolution Strategies](https://mlsanity.com/paper/2610.06993/) (2026) ## Abstract (arXiv) Reinforcement Learning with Verifiable Rewards (RLVR) offers a robust mechanism for enhancing mathematical reasoning in large models. However, we identify a systematic lack of emphasis on more challenging questions in existing methods from both algorithmic and data perspectives, despite their importance for refining underdeveloped capabilities. Algorithmically, widely used Group Relative Policy Optimization (GRPO) suffers from an implicit imbalance where the magnitude of policy updates is lower for harder questions. Data-wise, augmentation approaches primarily rephrase questions to enhance diversity without systematically increasing intrinsic difficulty. To address these issues, we propose a two-dual MathForge framework to improve mathematical reasoning by targeting harder questions from both perspectives, which comprises a Difficulty-Aware Group Policy Optimization (DGPO) algorithm and a Multi-Aspect Question Reformulation (MQR) strategy. Specifically, DGPO first rectifies the implicit imbalance in GRPO via difficulty-balanced group advantage estimation, and further prioritizes harder questions by difficulty-aware question-level weighting. Meanwhile, MQR reformulates questions across multiple aspects to increase difficulty while maintaining the original gold answer. Overall, MathForge forms a synergistic loop: MQR expands the data frontier, and DGPO effectively learns from the augmented data. Extensive experiments show that MathForge significantly outperforms existing methods on various mathematical reasoning tasks. The code and augmented data are all available at https://github.com/AMAP-ML/MathForge.