mlsanityData to 7 Oct 2026

Absolute Policy Optimization

arXiv 2310.13230 ↗First posted 20 Oct 2023cs.LG cs.AI cs.ROcited by 2

Not on the map. Why?

Abstract (arXiv)

In recent years, trust region on-policy reinforcement learning has achieved impressive results in addressing complex control tasks and gaming scenarios. However, contemporary state-of-the-art algorithms within this category primarily emphasize improvement in expected performance, lacking the ability to control over the worst-case performance outcomes. To address this limitation, we introduce a novel objective function, optimizing which leads to guaranteed monotonic improvement in the lower probability bound of performance with high confidence. Building upon this groundbreaking theoretical advancement, we further introduce a practical solution called Absolute Policy Optimization (APO). Our experiments demonstrate the effectiveness of our approach across challenging continuous control benchmark tasks and extend its applicability to mastering Atari games. Our findings reveal that APO as well as its efficient variation Proximal Absolute Policy Optimization (PAPO) significantly outperforms state-of-the-art policy gradient algorithms, resulting in substantial improvements in worst-case performance, as well as expected performance.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 52 of its 76 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

52 of 76 references.

References · 52

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  9. 2016OpenAI Gymcited by 1,684
  10. 2017Constrained Policy Optimizationcited by 928
  11. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  12. 2017A Distributional Perspective on Reinforcement Learningcited by 743
38 more
  1. 2019Solving Rubik's Cube with a Robot Handcited by 699
  2. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  3. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  4. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  5. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  6. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  7. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  8. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  9. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  10. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  11. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  12. 2018Stochastic Variance-Reduced Policy Gradientcited by 103
  13. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 87
  14. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  15. 2019Truly Proximal Policy Optimizationcited by 68
  16. 2019Worst Cases Policy Gradientscited by 52
  17. 2023State-wise Safe Reinforcement Learning: A Surveycited by 48
  18. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 39
  19. 2022You May Not Need Ratio Clipping in PPOcited by 19
  20. 2023AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learningcited by 16
  21. 2022Probabilistic Safeguard for Reinforcement Learning Using Safety Index Guided Gaussian Process Modelscited by 14
  22. 2023GUARD: A Safe Reinforcement Learning Benchmarkcited by 13
  23. 2022Persistently Feasible Robust Safe Control by Safety Index Synthesis and Convex Semi-Infinite Programmingcited by 11
  24. 2022Safety Index Synthesis via Sum-of-Squares Programmingcited by 11
  25. 2023Probabilistic Constraint for Safety-Critical Reinforcement Learningcited by 11
  26. 2019Compatible Natural Gradient Policy Searchcited by 10
  27. 2021Variance Penalized On-Policy and Off-Policy Actor-Criticcited by 10
  28. 2023State-wise Constrained Policy Optimizationcited by 9
  29. 2019Safe Adaptation with Multiplicative Uncertainties Using Robust Safe Set Algorithmcited by 8
  30. 2019Stochastic Variance Reduction for Deep Q-learningcited by 7
  31. 2020Experimental Evaluation of Human Motion Prediction: Toward Safe and Efficient Human Robot Collaborationcited by 7
  32. 2023On the Global Convergence of Risk-Averse Natural Policy Gradient Methods with Expected Conditional Risk Measurescited by 7
  33. 2017Stochastic Variance Reduction for Policy Gradient Estimationcited by 6
  34. 2021Provably Safe Tolerance Estimation for Robot Arms via Sum-of-Squares Programmingcited by 6
  35. 2023Learn With Imagination: Safe Set Guided State-wise Constrained Policy Optimizationcited by 4
  36. 2019Policy Optimization Through Approximate Importance Samplingcited by 3
  37. 2023Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimizationcited by 3
  38. 2020Proximal Policy Optimization Smoothed Algorithmcited by 2

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfactioncited by 0
  2. 2026ANO: Robust Policy Optimization via Bounded, Redescending Gain Fieldscited by 0