mlsanityData to 14 Oct 2026

MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems

arXiv 2509.22047 ↗First posted 26 Sep 2025cs.LGcited by 23

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Group Relative Policy Optimization (GRPO) has been shown to be an effective algorithm when an accurate reward model is available. However, such a highly reliable reward model is not available in many real-world tasks. In this paper, we particularly focus on multi-objective settings, in which we identify that GRPO is vulnerable to reward hacking, optimizing only one of the objectives at the cost of the others. To address this issue, we propose MO-GRPO, an extension of GRPO with a simple normalization method to reweight the reward functions automatically according to the variances of their values. We first show analytically that MO-GRPO ensures that all reward functions contribute evenly to the loss function while preserving the order of preferences, eliminating the need for manual tuning of the reward functions' scales. Then, we evaluate MO-GRPO experimentally in four domains: (i) the multi-armed bandits problem, (ii) simulated control task (Mo-Gymnasium), (iii) machine translation tasks on the WMT benchmark (En-Ja, En-Zh), and (iv) instruction following task. MO-GRPO achieves stable learning by evenly distributing correlations among the components of rewards, outperforming GRPO, showing MO-GRPO to be a promising algorithm for multi-objective reinforcement learning problems.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 44 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

29 of 44 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control5 Aug
  2. 2025RePO: Replay-Enhanced Policy Optimization11 Jun
  3. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment20 Feb
  7. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr

Other references · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2016Concrete Problems in AI Safetycited by 1,642
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  9. 2025Group Sequence Policy Optimizationcited by 835
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  11. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  12. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
10 more
  1. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 256
  2. 2024Reinforcement learningcited by 247
  3. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  4. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 214
  5. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  6. 2016Learning values across many orders of magnitudecited by 85
  7. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  8. 2021Return-based Scaling: Yet Another Normalisation Trick for Deep RLcited by 23
  9. 2023Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Trickscited by 7
  10. 2024An Open Multilingual System for Scoring Readability of Wikipediacited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 23.

23 papers.

Sort:

By area:

  1. 2026Faithful GRPO: Improving Visual Spatial Reasoning in Multimodal Language Models via Constrained Policy Optimizationcited by 6
  2. 2026The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Workscited by 5
  3. 2026On the Hidden Objective Biases of Group-based Reinforcement Learningcited by 3
  4. 2026SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Modelscited by 3
  5. 2026GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimizationcited by 3
  6. 2026Constrained Group Relative Policy Optimizationcited by 2
  7. 2026Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimizationcited by 2
  8. 2026What to Edit Next: Visually Aligned Image-Editing Follow-Up Suggestions in Conversational Systemscited by 2
  9. 2026IB-GRPO: Aligning LLM-based Learning Path Recommendation with Educational Objectives via Indicator-Based Group Relative Policy Optimizationcited by 1
  10. 2026PrAg-PO: Prompt Augmented Policy Optimization for Robust and Diverse Mathematical Reasoningcited by 1
  11. 2026TEMPO: Temporal Enforcement via Mode-Separated Policy Optimization for Trustworthy LLM Backtestingcited by 1
  12. 2026From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mixcited by 1
11 more
  1. 2026Signal Reshaping for GRPO in Weak-Feedback Agentic Code Repaircited by 0
  2. 2026LC-ERD: Mining Latent Logic for Self-Evolving Reasoning via Consistency-Regulated Reward Decompositioncited by 0
  3. 2026Multi-Stakeholder LLM Alignment: Decomposing Estimation from Aggregationcited by 0
  4. 2026CAT-Translate: Building Compact Open-Source Models for Japanese-English Translationcited by 0
  5. 2026Spurious Advantage Hidden in GRPOcited by 0
  6. 2026SteerDuplex: Steerable Duplex Speech Dialogue Modelscited by 0
  7. 2026SWE-MILE: Asynchronous Potential-Induced Milestone Credit Assignment for Long-Horizon Software Engineering Agentscited by 0
  8. 2026ORPG: Reconciling Multiple Reward Objectives through Objective-wise Policy Gradientscited by 0
  9. 2026CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learningcited by 0
  10. 2026PreciseCoder: Faithful and Generalizable Debugging Beyond Unit-Test Rewardscited by 0
  11. 2026AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training
  2. 2026MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following
  3. 2026GOPO: Policy Optimization using Ranked Rewards
  4. 2025GRPO is Secretly a Process Reward Model
  5. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
  6. 2025What is the Alignment Objective of GRPO?
  7. 2026STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hacking
  8. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments