mlsanityData to 14 Oct 2026

MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following

arXiv 2606.06058 ↗First posted 4 Jun 2026cs.LG cs.AI cs.CLcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards is ideal for multi-constraint instruction following, yet standard group-relative policy optimization (GRPO) becomes unstable under discrete, low-dispersion rewards, where within-group reward distributions are frequently homogeneous. We identify and formalize three pathologies of z-score group normalization in this regime: low-variance amplification, mean-centering blindness, and zero-variance collapse. To address them, we propose MDP-GRPO, which stabilizes learning through (1) multi-temperature sampling to increase reward dispersion, (2) dual-anchor advantages to restore gradients in homogeneous groups and stop mean-centering blindness, (3) prospect-theoretic shaping to bound updates and penalize violations based on Kahneman and Tversky's theory, and (4) asymmetric KL regularization. Evaluated on FollowBench, IFEval, and a curated multi-constraint dataset, MDP-GRPO outperforms standard GRPO, improving strict constraint satisfaction by up to 5.0% on Llama-3.2-3B. Our method also enables stable convergence with small group sizes while preserving general capabilities on MMLU and ARC.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 10 of its 29 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

10 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025NGRPO: Negative-enhanced Group Relative Policy Optimization23 Sep
  2. 2025MAPO: Mixed Advantage Policy Optimization23 Sep
  3. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun

Other references · 7

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  5. 2025Flexible and Efficient Grammar-Constrained Decodingcited by 43
  6. 2025Learning Safety Constraints for Large Language Modelscited by 13
  7. 2025Shaping Sparse Rewards in Reinforcement Learning: A Semi-supervised Approachcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recoverycited by 1
  2. 2026Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Constrained Group Relative Policy Optimization
  2. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
  3. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  4. 2026EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance
  5. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems
  6. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
  7. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training
  8. 2026Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Works