GRPO Training Dynamics for Small Language Models
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Group Relative Policy Optimization (GRPO) has emerged as a memory-efficient reinforcement fine-tuning (RFT) technique for reasoning-intensive tasks. How- ever, GRPO training dynamics on small language models (SLMs) remain poorly understood, limiting its reliable adoption and reproducibility in open and resource- constrained environments. In this work, we present a systematic study of GRPO fine-tuning for SLMs ranging from 1.5B to 7B parameters under a practical single- node 8xA100 compute budget. Our study spans multiple model families and reasoning domains, including mathematics, coding, and multiple-choice question answering (MCQ) in science. Across these settings, we analyze how group size affects policy convergence, training stability, and downstream benchmark per- formance. We further characterize tensor-level update dynamics during GRPO training and investigate whether the choice of LoRA target modules and layers can improve the performance of GRPO-tuned models. While our initial GRPO-tuned models outperform their base counterparts on approximately 80% of mathematical benchmark evaluations, they demonstrate limited capability on MCQ and code reasoning tasks. Guided by our mechanistic evaluations, we refined our LoRA and reward-shaping configurations to improve performance in latter domains. These findings provide practical guidance for GRPO training for SLMs.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 7 of its 54 references that are in the map. First posted on arXiv 30 Sep 2026.
Builds on
7 of 54 references.
Closest ancestors · 3
Same direction, newest first
- 2025On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization24 May
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 4
- 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
- 2025Tina: Tiny Reasoning Models via LoRAcited by 23
- 2026Mechanistic Interpretability for Large Language Model Alignment: Progress, Challenges, and Future Directionscited by 8
- 2025Hyperparameter Optimisation with Practical Interpretability and Explanation Methods in Probabilistic Curriculum Learningcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
- 2025Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Models
- 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
- 2025Multi-Layer GRPO: Enhancing Reasoning and Self-Correction in Large Language Models
- 2025Composing Policy Gradients and Prompt Optimization for Language Model Programs
- 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO
- 2025GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning
- 2026iGRPO: Self-Feedback-Driven LLM Reasoning