mlsanityData to 14 Oct 2026

Group Sequence Policy Optimization

arXiv 2507.18071 ↗First posted 24 Jul 2025cs.LG cs.AI cs.CLcited by 835

On the map: Mixture-of-experts → MoE in reinforcement learning and robotics

Abstract (arXiv)

This paper introduces Group Sequence Policy Optimization (GSPO), our stable, efficient, and performant reinforcement learning algorithm for training large language models. Unlike previous algorithms that adopt token-level importance ratios, GSPO defines the importance ratio based on sequence likelihood and performs sequence-level clipping, rewarding, and optimization. We demonstrate that GSPO achieves superior training efficiency and performance compared to the GRPO algorithm, notably stabilizes Mixture-of-Experts (MoE) RL training, and has the potential for simplifying the design of RL infrastructure. These merits of GSPO have contributed to the remarkable improvements in the latest Qwen3 models.

Key facts. Cited by 835 papers in the map (data through 14 Oct 2026). Placed in MoE in reinforcement learning and robotics, one of 12 directions in Mixture-of-experts. Builds on 4 of its 5 references that are in the map. First posted on arXiv 24 Jul 2025.

Builds on

4 of 5 references.

References · 4

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 196; 2026: 639.

835 papers.

Sort:

By area:

  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  2. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  3. 2025Qwen3-Omni Technical Reportcited by 497
  4. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  5. 2026Reinforcement Learning via Self-Distillationcited by 332
  6. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  7. 2026MiMo-V2-Flash Technical Reportcited by 191
  8. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  9. 2026Qwen3.5-Omni Technical Reportcited by 174
  10. 2025Qwen3Guard Technical Reportcited by 155
  11. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 146
  12. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
38 more
  1. 2025Soft Adaptive Policy Optimizationcited by 96
  2. 2026SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalizationcited by 94
  3. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 82
  4. 2025Tree Search for LLM Agent Reinforcement Learningcited by 79
  5. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 76
  6. 2025WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agentscited by 68
  7. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 68
  8. 2025Defeating the Training-Inference Mismatch via FP16cited by 66
  9. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 65
  10. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
  11. 2025Training-Free Group Relative Policy Optimizationcited by 55
  12. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 55
  13. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 54
  14. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 53
  15. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 53
  16. 2025FLEX: Continuous Agent Evolution via Forward Learning from Experiencecited by 52
  17. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 49
  18. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 49
  19. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 49
  20. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agentscited by 48
  21. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 47
  22. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  23. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 45
  24. 2026Dr. Zero: Self-Evolving Search Agents without Training Datacited by 44
  25. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 41
  26. 2025DCPO: Dynamic Clipping Policy Optimizationcited by 40
  27. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 40
  28. 2025Agentic Entropy-Balanced Policy Optimizationcited by 40
  29. 2025OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipecited by 37
  30. 2026MetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wildcited by 37
  31. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 37
  32. 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
  33. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  34. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 35
  35. 2026LongCat-Flash-Thinking-2601 Technical Reportcited by 34
  36. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimizationcited by 33
  37. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 33
  38. 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 33

50 most cited of 835 All 835 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
  2. 2025GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning
  3. 2026Soft Sequence Policy Optimization
  4. 2026Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR
  5. 2025Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Models
  6. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
  7. 2025Group-in-Group Policy Optimization for LLM Agent Training
  8. 2026GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models