mlsanityData to 14 Oct 2026

GVPO: Group Variance Policy Optimization for Large Language Model Post-Training

arXiv 2504.19599 ↗First posted 28 Apr 2025cs.AI cs.LGcited by 32

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Post-training plays a crucial role in refining and aligning large language models to meet specific tasks and human preferences. While recent advancements in post-training techniques, such as Group Relative Policy Optimization (GRPO), leverage increased sampling with relative reward scoring to achieve superior performance, these methods often suffer from training instability that limits their practical adoption. As a next step, we present Group Variance Policy Optimization (GVPO). GVPO incorporates the analytical solution to KL-constrained reward maximization directly into its gradient weights, ensuring alignment with the optimal policy. The method provides intuitive physical interpretations: its gradient mirrors the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) it guarantees a unique optimal solution, exactly the KL-constrained reward maximization objective, (2) it supports flexible sampling distributions that avoids on-policy and importance sampling limitations. By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 57 references that are in the map. First posted on arXiv 28 Apr 2025.

Builds on

37 of 57 references.

References · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2015Trust Region Policy Optimizationcited by 2,580
  11. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  12. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
25 more
  1. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  3. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  5. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  6. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  7. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  8. 2018Understanding the impact of entropy on policy optimizationcited by 176
  9. 2020Revisiting Fundamentals of Experience Replaycited by 175
  10. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  11. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 101
  12. 2024Explaining Length Bias in LLM-Based Preference Evaluationscited by 49
  13. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understandingcited by 34
  14. 2025SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reductioncited by 24
  15. 2024Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learningcited by 21
  16. 2025LLM-powered Multi-agent Framework for Goal-oriented Learning in Intelligent Tutoring Systemcited by 21
  17. 2025Mark Your LLM: Detecting the Misuse of Open-Source Large Language Models via Watermarkingcited by 19
  18. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  19. 2024Learning from negative feedback, or positive feedback or bothcited by 13
  20. 2024Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Modelcited by 9
  21. 2025Beyond the Next Token: Towards Prompt-Robust Zero-Shot Classification via Efficient Multi-Token Predictioncited by 7
  22. 2022Asymptotic comparison of identifying constraints for Bradley-Terry modelscited by 6
  23. 2024Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Modelscited by 5
  24. 2024Optimized Cost Per Click in Online Advertising: A Theoretical Analysiscited by 3
  25. 2023The Impact of Generative Artificial Intelligence on Market Equilibrium: Evidence from a Natural Experimentcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 19.

32 papers.

Sort:

By area:

  1. 2025Geometric-Mean Policy Optimizationcited by 111
  2. 2025BusterX++: Towards Unified Cross-Modal AI-Generated Content Detection and Explanation with MLLMcited by 29
  3. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 27
  4. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 19
  5. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
  6. 2025AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoningcited by 11
  7. 2025ReDit: Reward Dithering for Improved LLM Policy Optimizationcited by 8
  8. 2025DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Datacited by 7
  9. 2026Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimizationcited by 7
  10. 2026PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learningcited by 7
  11. 2025You only need 4 extra tokens: Synergistic Test-time Adaptation for LLMscited by 6
  12. 2026Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoningcited by 6
20 more
  1. 2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 5
  2. 2026On the Plasticity and Stability for Post-Training Large Language Modelscited by 4
  3. 2025Via Score to Performance: Efficient Human-Controllable Long Song Generation with Bar-Level Symbolic Notationcited by 3
  4. 2025Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integrationcited by 3
  5. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimizationcited by 3
  6. 2026SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learningcited by 3
  7. 2026Joint Continual Learning of Local Language Models and Cloud Offloading Decisions with Budget Constraintscited by 2
  8. 2025SPR-128K: A New Benchmark for Spatial Plausibility Reasoning with Multimodal Large Language Modelscited by 1
  9. 2025Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Studycited by 1
  10. 2026Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoningcited by 1
  11. 2026Online Causal Kalman Filtering for Stable and Effective Policy Optimizationcited by 1
  12. 2026Diffusion Alignment Beyond KL: Variance Minimisation as Effective Policy Optimisercited by 1
  13. 2026Quality-constrained Entropy Maximization Policy Optimization for LLM Diversitycited by 1
  14. 2026All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Modelscited by 1
  15. 2026Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradientscited by 1
  16. 2026F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Rankingcited by 1
  17. 2026PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMscited by 0
  18. 2026Tracking the Moving Frontier: Long-Short Term Advantage Estimatorcited by 0
  19. 2026GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillationcited by 0
  20. 2026Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
  2. 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
  3. 2025Group Sequence Policy Optimization
  4. 2025Group Causal Policy Optimization for Post-Training Large Language Models
  5. 2025Accelerating RLHF Training with Reward Variance Increase
  6. 2026Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervals
  7. 2025GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning
  8. 2025Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach