# Prefix Grouper: Efficient GRPO Training through Shared-Prefix Forward

arXiv: https://arxiv.org/abs/2506.05433 · HTML: https://mlsanity.com/paper/2506.05433/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 8 of its 8 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 8 of 8)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2503.06749/) (2025)
- [VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model](https://mlsanity.com/paper/2504.07615/) (2025)
- [Video-R1: Reinforcing Video Reasoning in MLLMs](https://mlsanity.com/paper/2503.21776/) (2025)
- [VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2504.06958/) (2025)
- [Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding](https://mlsanity.com/paper/2503.13377/) (2025)

## Built on by (most cited first, 9 of 9)

- [SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts](https://mlsanity.com/paper/2509.23232/) (2025)
- [Tree Training: Accelerating Agentic LLMs Training via Shared Prefix Reuse](https://mlsanity.com/paper/2511.00413/) (2025)
- [DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts](https://mlsanity.com/paper/2605.15422/) (2026)
- [Faster Synchronous On-Policy RL via Straggler-Aware Group Sizing](https://mlsanity.com/paper/2606.02218/) (2026)
- [LaV-CoT: Language-Aware Visual CoT with Multi-Aspect Reward Optimization for Real-World Multilingual VQA](https://mlsanity.com/paper/2509.10026/) (2025)
- [Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMs](https://mlsanity.com/paper/2601.21476/) (2026)
- [Consensus Group Relative Policy Optimization for Text Generation](https://mlsanity.com/paper/2602.03102/) (2026)
- [Are complicated loss functions necessary for teaching LLMs to reason?](https://mlsanity.com/paper/2603.18756/) (2026)
- [Rollplex: Cross-Phase GPU Spatial Sharing for Vision Language Model Post-Training](https://mlsanity.com/paper/2608.14498/) (2026)

## Abstract (arXiv)

Group Relative Policy Optimization (GRPO) enhances policy learning by computing gradients from relative comparisons among candidate outputs that share a common input prefix. Despite its effectiveness, GRPO introduces substantial computational overhead when processing long shared prefixes, which must be redundantly encoded for each group member. This inefficiency becomes a major scalability bottleneck in long-context learning scenarios. We propose Prefix Grouper, an efficient GRPO training algorithm that eliminates redundant prefix computation via a Shared-Prefix Forward strategy. In particular, by restructuring self-attention into two parts, our method enables the shared prefix to be encoded only once, while preserving full differentiability and compatibility with end-to-end training. We provide both theoretical and empirical evidence that Prefix Grouper is training-equivalent to standard GRPO: it yields identical forward outputs and backward gradients, ensuring that the optimization dynamics and final policy performance remain unchanged. Empirically, our experiments confirm that Prefix Grouper achieves consistent results while significantly reducing the computational cost of training, particularly in long-prefix scenarios. The proposed method is fully plug-and-play: it is compatible with existing GRPO-based architectures and can be seamlessly integrated into current training pipelines as a drop-in replacement, requiring no structural modifications and only minimal changes to input construction and attention computation. Prefix Grouper enables the use of larger group sizes under the same computational budget, thereby improving the scalability of GRPO to more complex tasks and larger models. Code is now available at https://github.com/johncaged/PrefixGrouper