mlsanityData to 14 Oct 2026

Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning

arXiv 2510.19807 ↗First posted 22 Oct 2025cs.CL cs.AI cs.LGcited by 52

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning from verifiable rewards has emerged as a powerful technique for enhancing the complex reasoning abilities of Large Language Models (LLMs). However, these methods are fundamentally constrained by the ''learning cliff'' phenomenon: when faced with problems far beyond their current capabilities, models consistently fail, yielding a persistent zero-reward signal. In policy optimization algorithms like GRPO, this collapses the advantage calculation to zero, rendering these difficult problems invisible to the learning gradient and stalling progress. To overcome this, we introduce Scaf-GRPO (Scaffolded Group Relative Policy Optimization), a progressive training framework that strategically provides minimal guidance only when a model's independent learning has plateaued. The framework first diagnoses learning stagnation and then intervenes by injecting tiered in-prompt hints, ranging from abstract concepts to concrete steps, enabling the model to construct a valid solution by itself. Extensive experiments on challenging mathematics benchmarks demonstrate Scaf-GRPO's effectiveness, boosting the pass@1 score of the Qwen2.5-Math-7B model on the AIME24 benchmark by a relative 44.3% over a vanilla GRPO baseline. This result demonstrates our framework provides a robust and effective methodology for unlocking a model's ability to solve problems previously beyond its reach, a critical step towards extending the frontier of autonomous reasoning in LLM.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 37 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

24 of 37 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR21 Jul
  2. 2025StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason3 Jul
  3. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling2 Jul
  4. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning20 Jun
  5. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  6. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  7. 2025Learning to Reason under Off-Policy Guidance21 Apr
  8. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  10. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025Process Reinforcement through Implicit Rewards3 Feb
1 more
  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan

Other references · 11

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2025s1: Simple test-time scalingcited by 1,341
  4. 2024OpenAI o1 System Cardcited by 1,244
  5. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  6. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  7. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 384
  8. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  9. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  10. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 164
  11. 2025ReCUT: Balancing Reasoning Length and Accuracy in LLMs via Stepwise Trails and Preference Optimizationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 51.

52 papers.

Sort:

By area:

  1. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 42
  2. 2026Self-Hinting Language Models Enhance Reinforcement Learningcited by 20
  3. 2026Learning to Hint for Reinforcement Learningcited by 12
  4. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  5. 2026Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Taskscited by 9
  6. 2026No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learningcited by 8
  7. 2026KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidancecited by 8
  8. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
  9. 2026Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Informationcited by 7
  10. 2026The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Workscited by 5
  11. 2026Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Modelscited by 4
  12. 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuningcited by 3
38 more
  1. 2026Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidancecited by 3
  2. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimizationcited by 3
  3. 2026Max Out GRPO Signal: Adaptive Trace Prefix Control for Hard Reasoning Problemscited by 3
  4. 2026Efficient Paths and Dense Rewards: Probabilistic Flow Reasoning for Large Language Modelscited by 2
  5. 2026S-GRPO: Unified Post-Training for Large Vision-Language Modelscited by 2
  6. 2026From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoningcited by 2
  7. 2026Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policycited by 2
  8. 2026KEPO: Knowledge-Enhanced Preference Optimization for Multimodal Reasoning with Applications to Medical VQAcited by 1
  9. 2026Implicit Strategic Optimization: Rethinking Long-Horizon Decision-Making in Adversarial Poker Environmentscited by 1
  10. 2026Internalizing Meta-Experience into Memory for Guided Reinforcement Learning in Large Language Modelscited by 1
  11. 2026Nonsense Helps: Prompt Space Perturbation Broadens Reasoning Explorationcited by 1
  12. 2026Selective Off-Policy Reference Tuning with Plan Guidancecited by 1
  13. 2026Seirenes: Adversarial Self-Play with Evolving Distractions for LLM Reasoningcited by 1
  14. 2026Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMscited by 1
  15. 2026Smaller Models are Natural Explorers for Policy-Level Diversity in GRPOcited by 1
  16. 2026Learning from Unreachable Rewards: Hint-Conditioned Reinforcement Learning for Generative Recommendationcited by 1
  17. 2026Agent-G2: Gaussian Guidance for Agentic Reinforcement Learningcited by 1
  18. 2026The Model Knows Another Way: Strategy Switching for Effective RLVR Explorationcited by 1
  19. 2026RLTL;DR: Self-improvement by Internalizing Self-generated Feedbackcited by 1
  20. 2025Multi-Reward GRPO Fine-Tuning for De-biasing Large Language Models: A Study Based on Chinese-Context Discrimination Datacited by 0
  21. 2026GRASP LoRA: GRPO Guided Adapter Sparsity Policy for Cross Lingual Transfercited by 0
  22. 2026Constraint-Rectified Training for Efficient Chain-of-Thoughtcited by 0
  23. 2026Mitigating Distribution Sharpening in Math RLVR via Distribution-Aligned Hint Synthesis and Backward Hint Annealingcited by 0
  24. 2026CRPO: Character-centric Group Relative Policy Optimization for Role-aware Reasoning in Role-playing Agentscited by 0
  25. 2026Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimizationcited by 0
  26. 2026TD-Grokking: Learning from Zero-Reward Problems by Training-Time Decompositioncited by 0
  27. 2026LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Promptscited by 0
  28. 2026Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Tracescited by 0
  29. 2026ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interactioncited by 0
  30. 2026Dynamic Important Example Mining for Reinforcement Finetuningcited by 0
  31. 2026When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluationcited by 0
  32. 2026Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Taskscited by 0
  33. 2026Feyospace-v1: How the Cyber Mercury Seven Trained Frontier Cyber Modelscited by 0
  34. 2026RAISE: Reinforcing Access Control Policy Synthesis in LLMs via Symbolic Evaluationcited by 0
  35. 2026Teach to Learn: Hint Annealing for Self-improving LLM Reasoningcited by 0
  36. 2026Agentic Tool-Augmented Reasoning for Explainable Image Forgery Detectioncited by 0
  37. 2026Learning from Evolving Errors: Adaptive Iterative Repair for On-Policy Distillationcited by 0
  38. 2026Understanding Enrichment in Reinforcement Learningcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026iGRPO: Self-Feedback-Driven LLM Reasoning
  2. 2026GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
  3. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  4. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?
  5. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
  6. 2026Are complicated loss functions necessary for teaching LLMs to reason?
  7. 2025Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting
  8. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models