# Rethinking the Trust Region in LLM Reinforcement Learning

arXiv: https://arxiv.org/abs/2602.04879 · HTML: https://mlsanity.com/paper/2602.04879/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 38 of its 53 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 30 of 38)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [UltraFeedback: Boosting Language Models with Scaled AI Feedback](https://mlsanity.com/paper/2310.01377/) (2023)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024)
- [MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention](https://mlsanity.com/paper/2506.13585/) (2025)
- [Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data](https://mlsanity.com/paper/2404.14367/) (2024)
- [OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling](https://mlsanity.com/paper/2506.20512/) (2025)
- [The Art of Scaling Reinforcement Learning Compute for LLMs](https://mlsanity.com/paper/2510.13786/) (2025)
- [Learning Dynamics of LLM Finetuning](https://mlsanity.com/paper/2407.10490/) (2024)
- [Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers](https://mlsanity.com/paper/2510.11370/) (2025)
- [Truly Proximal Policy Optimization](https://mlsanity.com/paper/1903.07940/) (2019)
- [Defeating the Training-Inference Mismatch via FP16](https://mlsanity.com/paper/2510.26788/) (2025)
- [Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model](https://mlsanity.com/paper/2510.18855/) (2025)
- [Stabilizing Reinforcement Learning with LLMs: Formulation and Practices](https://mlsanity.com/paper/2512.01374/) (2025)

## Built on by (most cited first, 30 of 37)

- [T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks](https://mlsanity.com/paper/2609.11042/) (2026)
- [Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models](https://mlsanity.com/paper/2606.11025/) (2026)
- [Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents](https://mlsanity.com/paper/2606.26080/) (2026)
- [Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction](https://mlsanity.com/paper/2605.12070/) (2026)
- [Rethinking the Divergence Regularization in LLM RL](https://mlsanity.com/paper/2606.09821/) (2026)
- [Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning](https://mlsanity.com/paper/2607.18722/) (2026)
- [Tail-Aware Top-k On-Policy Distillation](https://mlsanity.com/paper/2608.14728/) (2026)
- [ZAYA1-8B Technical Report](https://mlsanity.com/paper/2605.05365/) (2026)
- [Learning from Your Own Mistakes: Constructing Learnable Micro-Reflective Trajectories for Self-Distillation](https://mlsanity.com/paper/2606.18844/) (2026)
- [Best Practice Critic Optimization](https://mlsanity.com/paper/2608.23566/) (2026)
- [FrogNano: Training a 4B Coding Agent via Online Task Synthesis](https://mlsanity.com/paper/2609.07925/) (2026)
- [Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning](https://mlsanity.com/paper/2606.10968/) (2026)
- [Score Centering Stabilizes Off-policy Reinforcement Learning](https://mlsanity.com/paper/2609.20807/) (2026)
- [Policy Improvement Reinforcement Learning](https://mlsanity.com/paper/2604.00860/) (2026)
- [Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment](https://mlsanity.com/paper/2607.04728/) (2026)
- [Learning Generalizable Behaviors for Terminal Agents](https://mlsanity.com/paper/2608.22631/) (2026)
- [Apodex 1.1: Scaling Agentic Intelligence for Complex Work](https://mlsanity.com/paper/2608.23283/) (2026)
- [Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates](https://mlsanity.com/paper/2605.20005/) (2026)
- [Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2605.20865/) (2026)
- [Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning](https://mlsanity.com/paper/2606.13680/) (2026)
- [SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation](https://mlsanity.com/paper/2608.09271/) (2026)
- [CoTrace: Data Recipes for Training Terminal Agents with Harness-Model Co-Evolution](https://mlsanity.com/paper/2610.10426/) (2026)
- [Bounded Ratio Reinforcement Learning](https://mlsanity.com/paper/2604.18578/) (2026)
- [UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma](https://mlsanity.com/paper/2607.06987/) (2026)
- [Predictive Divergence Masks for LLM RL](https://mlsanity.com/paper/2607.10848/) (2026)
- [Performance Foundations of Parallel & Distributed Reasoning Language Models](https://mlsanity.com/paper/2608.27046/) (2026)
- [Train What You Deploy:Token-Faithful Post-Training of a Production Coding](https://mlsanity.com/paper/2609.04678/) (2026)
- [Training Large Language Models for Small-Molecule Design with Synthetic Task Scaling](https://mlsanity.com/paper/2609.04735/) (2026)
- [Bellman Policy Optimization](https://mlsanity.com/paper/2609.15987/) (2026)
- [Towards Full Pipeline FP8 Reinforcement Learning for LLMs](https://mlsanity.com/paper/2609.22870/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm. Despite its ubiquity, we argue that the core ratio clipping mechanism in PPO is structurally ill-suited for the large vocabularies inherent to LLMs. PPO constrains policy updates based on the probability ratio of sampled tokens, which serves as a noisy single-sample Monte Carlo estimate of the true policy divergence. This creates a sub-optimal learning dynamic: updates to low-probability tokens are aggressively over-penalized, while potentially catastrophic shifts in high-probability tokens are under-constrained, leading to training inefficiency and instability. To address this, we propose Divergence Proximal Policy Optimization (DPPO), which substitutes heuristic clipping with a more principled constraint based on a direct estimate of policy divergence (e.g., Total Variation or KL). To avoid huge memory footprint, we introduce the efficient Binary and Top-K approximations to capture the essential divergence with negligible overhead. Extensive empirical evaluations demonstrate that DPPO achieves superior training stability and efficiency compared to existing methods, offering a more robust foundation for RL-based LLM fine-tuning. Our code is available at https://github.com/sail-sg/Stable-RL.