mlsanityData to 14 Oct 2026

ISOPO: Proximal policy gradients without pi-old

arXiv 2512.23353 ↗First posted 29 Dec 2025cs.LGcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

This note introduces Isometric Policy Optimization (ISOPO), an efficient method to approximate the natural policy gradient in a single gradient step. In comparison, existing proximal policy methods such as GRPO or CISPO use multiple gradient steps with variants of importance ratio clipping to approximate a natural gradient step relative to a reference policy. In its simplest form, ISOPO normalizes the log-probability gradient of each sequence in the Fisher metric before contracting with the advantages. Another variant of ISOPO transforms the microbatch advantages based on the neural tangent kernel in each layer. ISOPO applies this transformation layer-wise in a single backward pass and can be implemented with negligible computational overhead compared to vanilla REINFORCE.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 19 references that are in the map. First posted on arXiv 29 Dec 2025.

Builds on

15 of 19 references.

References · 15

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  7. 2025Group Sequence Policy Optimizationcited by 835
  8. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  9. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  10. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  11. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  12. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
3 more
  1. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  2. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMscited by 44
  3. 2025On-Policy RL with Optimal Reward Baselinecited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026PROMA: Projected Microbatch Accumulation for Reference-Free Proximal Policy Updatescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Hybrid Group Relative Policy Optimization: A Multi-Sample Approach to Enhancing Policy Optimization
  2. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback
  3. 2025Symmetric Behavior Regularized Policy Optimization
  4. 2026Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients
  5. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
  6. 2025GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping
  7. 2025Improving DAPO from a Mixed-Policy Perspective
  8. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments