mlsanityData to 7 Oct 2026

Simple Policy Optimization

arXiv 2401.16025 ↗First posted 29 Jan 2024cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Model-free reinforcement learning algorithms have seen remarkable progress, but key challenges remain. Trust Region Policy Optimization (TRPO) is known for ensuring monotonic policy improvement through conservative updates within a trust region, backed by strong theoretical guarantees. However, its reliance on complex second-order optimization limits its practical efficiency. Proximal Policy Optimization (PPO) addresses this by simplifying TRPO's approach using ratio clipping, improving efficiency but sacrificing some theoretical robustness. This raises a natural question: Can we combine the strengths of both methods? In this paper, we introduce Simple Policy Optimization (SPO), a novel unconstrained first-order algorithm. By slightly modifying the policy loss used in PPO, SPO can achieve the best of both worlds. Our new objective improves upon ratio clipping, offering stronger theoretical properties and better constraining the probability ratio within the trust region. Empirical results demonstrate that SPO outperforms PPO with a simple implementation, particularly for training large, complex network architectures end-to-end.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 0 of its 1 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

0 of 1 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 26.

31 papers.

Sort:

By area:

  1. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  2. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 32
  3. 2026Flow Policy Gradients for Robot Controlcited by 14
  4. 2025Relative Entropy Pathwise Policy Optimizationcited by 6
  5. 2026Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuningcited by 6
  6. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  7. 2026Rethinking the Divergence Regularization in LLM RLcited by 5
  8. 2025PPO in the Fisher-Rao geometrycited by 4
  9. 2026Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Modelscited by 4
  10. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 4
  11. 2026Trust Region Policy Distillationcited by 2
  12. 2025Coordinated Strategies in Realistic Air Combat by Hierarchical Multi-Agent Reinforcement Learningcited by 1
19 more
  1. 2026RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learningcited by 1
  2. 2026Align and Filter: Improving Performance in Asynchronous On-Policy RLcited by 1
  3. 2025Representation Convergence: Mutual Distillation is Secretly a Form of Regularizationcited by 0
  4. 2026Rethinking Policy Diversity in Ensemble Policy Gradient in Large-Scale Reinforcement Learningcited by 0
  5. 2026Bounded Ratio Reinforcement Learningcited by 0
  6. 2026Do Not Imitate, Reinforce: Iterative Classification via Belief Refinementcited by 0
  7. 2026ANO: Robust Policy Optimization via Bounded, Redescending Gain Fieldscited by 0
  8. 2026Rethinking Ratio-Based Trust Regions for Policy Optimization in Multi-Agent Reinforcement Learningcited by 0
  9. 2026Ratio-Variance Regularized Policy Optimizationcited by 0
  10. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitionscited by 0
  11. 2026PACT: Self-Evolving Physical Safety Alignment for Diffusion Policies in Embodied Manipulationcited by 0
  12. 2026Reinforcement Learning for Flow-Matching Policies with Density Transportcited by 0
  13. 2026Trust-Region Diffusion Policies for Massively Parallel On-Policy RLcited by 0
  14. 2026KLip-PPO: A per-sample KL perspective on PPO-Clipcited by 0
  15. 2026Smart charging of large fleets of Electric Vehicles: Independent Multi-Agent Reinforcement Learning approachescited by 0
  16. 2026Predictive Divergence Masks for LLM RLcited by 0
  17. 2026OR Else: A Differentiable Trust Region for Policy Optimizationcited by 0
  18. 2026What Does a ProcGen Generalization Gap Measure? Action Rules, Residual Entropy, and the Missing Random Floorcited by 0
  19. 2026CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learningcited by 0