mlsanityData to 7 Oct 2026

Limits of Difficulty Scaling: Hard Samples Yield Diminishing Returns in GRPO-Tuned SLMs

arXiv 2604.06298 ↗First posted 7 Apr 2026cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent alignment work on Large Language Models (LLMs) suggests preference optimization can improve reasoning by shifting probability mass toward better solutions. We test this claim in a resource-constrained setting by applying GRPO with LoRA to SLMs (up to 3B) for math reasoning on GSM8K and MATH datasets with difficulty-stratified analyses. As problem difficulty increases, accuracy plateaus, revealing a capacity boundary: GRPO primarily reshapes output preferences without reliably improving hardest-tier solving. Consistent with this, training GRPO only on lower-difficulty problems matches full-dataset accuracy across difficulty tiers while using only ~45% training steps, indicating diminishing returns from harder samples in this regime. We also find a cross-dataset generalization effect: GSM8K-trained GRPO achieves higher accuracy on the numeric subset of MATH than MATH-trained GRPO, exceeding it by ~5% at 1.5B and by ~3% at 3B. We show that the best achievable gains depend strongly on the base model's prior reasoning competence and the dataset's difficulty profile.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 15 of its 19 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

15 of 19 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning7 Jun
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  11. 2023Textbooks Are All You Needcited by 561
  12. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 40
1 more
  1. 2024Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalizationcited by 24

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models
  2. 2026GRPO Training Dynamics for Small Language Models
  3. 2026The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning
  4. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?
  5. 2026DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantage
  6. 2026Are complicated loss functions necessary for teaching LLMs to reason?
  7. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening
  8. 2026MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting