mlsanityData to 14 Oct 2026

Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgets

arXiv 2508.14094 ↗First posted 15 Aug 2025cs.LG cs.AIcited by 17

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Collecting high-quality training examples for language model fine-tuning is expensive, with practical budgets limiting the amount of data that can be procured. We investigate whether example difficulty affects GRPO training effectiveness by comparing selection strategies (easy, medium, hard, random) across multiple models and reasoning tasks. Training on the hardest 10% of examples (those where the base model fails most often) yields dramatic performance gains up to 47%, while easy examples produce minimal improvements of 3-15%. This occurs because GRPO requires outcome variance to generate learning signals; hard examples maintain mixed success/failure outcomes throughout training while easy examples quickly converge to consistent success, eliminating learning opportunities. Moreover, models trained on hard examples show superior out-of-distribution generalization, with only hard-trained models achieving meaningful gains on the AIME2025 benchmark. Our findings provide clear guidance: when budget-constrained, prioritize collecting and annotating examples where your base model struggles, as these drive nearly all learning value in GRPO fine-tuning

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 25 references that are in the map. First posted on arXiv 15 Aug 2025.

Builds on

18 of 25 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay5 Jun
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025Process Reinforcement through Implicit Rewards3 Feb
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2024Phi-4 Technical Reportcited by 711
  8. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  9. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  10. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 286
  11. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 274
  12. 2025DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Trainingcited by 18
2 more
  1. 2025SPARQ: Synthetic Problem Generation for Reasoning via Quality-Diversity Algorithmscited by 7
  2. 2025Understanding Trade offs When Conditioning Synthetic Datacited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 13.

17 papers.

Sort:

By area:

  1. 2025BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuningcited by 14
  2. 2025MAPO: Mixed Advantage Policy Optimizationcited by 11
  3. 2026Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptationcited by 7
  4. 2025Revisiting Generalization Across Difficulty Levels: It's Not So Easycited by 5
  5. 2026DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoningcited by 3
  6. 2026LLM Reasoning with Process Rewards for Outcome-Guided Stepscited by 3
  7. 2026Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recoverycited by 1
  8. 2025ToMPO: Training LLM Strategic Decision Making from a Multi-Agent Perspectivecited by 0
  9. 2026Rethinking Reinforcement fine-tuning of LLMs: A Multi-armed Bandit Learning Perspectivecited by 0
  10. 2026Rethinking Easy-to-Hard: Limits of Curriculum Learning in Post-Training for Deductive Reasoningcited by 0
  11. 2026Adaptive Loss Balancing for Noise-Robust GRPO in Generative Recommendationcited by 0
  12. 2026Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHFcited by 0
5 more
  1. 2026Selective Left-Shift: Turning Test-Time Compute and Difficulty-based Curation into Training Data for Low-Resource Code Generationcited by 0
  2. 2026Wnuan: Staged Post-Training for Question Answering over Proprietary Enterprise Knowledgecited by 0
  3. 2026Unlocking the Unsolvable: Teacher-Guided Curriculum for Data-Efficient RLVRcited by 0
  4. 2026OmniFysics-Nano-V2 Technical Report: Understanding the Physical World Across Modalitiescited by 0
  5. 2026DART-ES: Difficulty-Aware Reweighting and Targeted Replay for Fine-Tuning LLMs with Evolution Strategiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Limits of Difficulty Scaling: Hard Samples Yield Diminishing Returns in GRPO-Tuned SLMs
  2. 2026GRPO Training Dynamics for Small Language Models
  3. 2026Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPO
  4. 2026A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO
  5. 2022Assessing Out-of-Domain Language Model Performance from Few Examples
  6. 2026On The Effectiveness-Fluency Trade-Off In LLM Conditioning: A Systematic Study
  7. 2025Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learning
  8. 2024The Unreasonable Effectiveness of Easy Training Data for Hard Tasks