Beyond KL Divergence: Policy Optimization with Flexible Bregman Divergences for LLM Reasoning
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Policy optimization methods like Group Relative Policy Optimization (GRPO) and its variants have achieved strong results on mathematical reasoning and code generation tasks. Despite extensive exploration of reward processing strategies and training dynamics, all existing group-based methods exclusively use KL divergence for policy regularization, leaving the choice of divergence function unexplored. We introduce Group-Based Mirror Policy Optimization (GBMPO), a framework that extends group-based policy optimization to flexible Bregman divergences, including hand-designed alternatives (L2 in probability space) and learned neural mirror maps. On GSM8K mathematical reasoning, hand-designed ProbL2-GRPO achieves 86.7% accuracy, improving +5.5 points over the Dr. GRPO baseline. On MBPP code generation, neural mirror maps reach 60.1-60.8% pass@1, with random initialization already capturing most of the benefit. While evolutionary strategies meta-learning provides marginal accuracy improvements, its primary value lies in variance reduction (±0.2 versus ±0.6) and efficiency gains (15% shorter responses on MBPP), suggesting that random initialization of neural mirror maps is sufficient for most practical applications. These results establish divergence choice as a critical, previously unexplored design dimension in group-based policy optimization for LLM reasoning.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 32 references that are in the map. First posted on arXiv 4 Feb 2026.
Builds on
14 of 32 references.
Closest ancestors · 8
Same direction, newest first
- 2025G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance18 Aug
- 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control5 Aug
- 2025TreeRPO: Tree Relative Policy Optimization5 Jun
- 2025Multi-Layer GRPO: Enhancing Reasoning and Self-Correction in Large Language Models5 Jun
- 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training28 May
- 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
- 2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization24 Dec
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 6
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2025Group Sequence Policy Optimizationcited by 835
- 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Breaking textitWinner-Takes-All: Cooperative Policy Optimization Improves Diverse LLM Reasoning
- 2025Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting
- 2026Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization
- 2026Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO
- 2025Adaptive Group Policy Optimization: Towards Stable Training and Token-Efficient Reasoning
- 2026Calibration-Aware Policy Optimization for Reasoning LLMs
- 2026Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence
- 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization