Unified Policy Optimization for Continuous-action Reinforcement Learning in Non-stationary Tasks and Games
Not on the map. Why?
Abstract (arXiv)
This paper addresses policy learning in non-stationary environments and games with continuous actions. Rather than the classical reward maximization mechanism, inspired by the ideas of follow-the-regularized-leader (FTRL) and mirror descent (MD) update, we propose a no-regret style reinforcement learning algorithm PORL for continuous action tasks. We prove that PORL has a last-iterate convergence guarantee, which is important for adversarial and cooperative games. Empirical studies show that, in stationary environments such as MuJoCo locomotion controlling tasks, PORL performs equally well as, if not better than, the soft actor-critic (SAC) algorithm; in non-stationary environments including dynamical environments, adversarial training, and competitive games, PORL is superior to SAC in both a better final policy performance and a more stable training process.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 25 of its 45 references that are in the map. First posted on arXiv 19 Aug 2022.
Builds on
25 of 45 references.
References · 25
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2022Generative Adversarial Networkscited by 5,627
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2015Trust Region Policy Optimizationcited by 2,580
- 2016Generative Adversarial Imitation Learningcited by 1,597
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
- 2019Introduction to Online Convex Optimizationcited by 803
- 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
- 2017Emergence of Grounded Compositional Language in Multi-Agent Populationscited by 263
- 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 158
- 2015Fast Convergence of Regularized Learning in Gamescited by 150
13 more
- 2018Last-Iterate Convergence: Zero-Sum Games and Constrained Min-Max Optimizationcited by 98
- 2021Fast Policy Extragradient Methods for Competitive Games with Entropy Regularizationcited by 63
- 2020From Poincar'e Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularizationcited by 61
- 2020Hedging in games: Faster convergence of external and swap regretscited by 57
- 2020Faster Game Solving via Predictive Blackwell Approachability: Connecting Regret Matching and Mirror Descentcited by 57
- 2018Actor-Critic Policy Optimization in Partially Observable Multiagent Environmentscited by 35
- 2021Last-iterate Convergence in Extensive-Form Gamescited by 28
- 2022Policy Optimization for Markov Games: Unified Framework and Faster Convergencecited by 16
- 2017Regret Minimization for Partially Observable Deep Reinforcement Learningcited by 12
- 2014A Unified View of Large-scale Zero-sum Equilibrium Computationcited by 7
- 2018Fictitious GAN: Training GANs with Historical Modelscited by 7
- 2021DO-GAN: A Double Oracle Framework for Generative Adversarial Networkscited by 6
- 2019Entropy-Regularized Stochastic Gamescited by 5
Built on by
Citing papers per year: 2023: 1.
1 paper.