PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithm
Not on the map. Why?
Abstract (arXiv)
In this paper, we propose the first fully push-forward-based distributional reinforcement learning algorithm, named PACER, which consists of a distributional critic, a stochastic actor and a sample-based encourager. Specifically, the push-forward operator is leveraged in both the critic and actor to model the return distributions and stochastic policies respectively, enabling them with equal modeling capability and thus enhancing the synergetic performance. Since it is infeasible to obtain the density function of the push-forward policies, novel sample-based regularizers are integrated in the encourager to incentivize efficient exploration and alleviate the risk of trapping into local optima. Moreover, a sample-based stochastic utility value policy gradient is established for the push-forward policy update, which circumvents the explicit demand of the policy density function in existing REINFORCE-based stochastic policy gradient. As a result, PACER fully utilizes the modeling capability of the push-forward operator and is able to explore a broader class of the policy space, compared with limited policy classes used in existing distributional actor critic algorithms (i.e. Gaussians). We validate the critical role of each component in our algorithm with extensive empirical studies. Experimental results demonstrate the superiority of our algorithm over the state-of-the-art.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 45 of its 78 references that are in the map. First posted on arXiv 11 Jun 2023.
Builds on
45 of 78 references.
References · 45
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
- 2015Variational Inference with Normalizing Flowscited by 2,170
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
- 2016OpenAI Gymcited by 1,736
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
- 2018Exploration by Random Network Distillationcited by 993
- 2018Computational Optimal Transportcited by 780
- 2017A Distributional Perspective on Reinforcement Learningcited by 765
- 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
- 2017Generative Adversarial Networks: An Overviewcited by 590
33 more
- 2014Semi-Supervised Learning with Deep Generative Modelscited by 496
- 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
- 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
- 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
- 2018Distributed Distributional Deterministic Policy Gradientscited by 199
- 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
- 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
- 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
- 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
- 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
- 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 141
- 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
- 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 85
- 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
- 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 67
- 2021Risk-Averse Offline Reinforcement Learningcited by 65
- 2019Distributional Reinforcement Learning for Efficient Explorationcited by 60
- 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
- 2019Leveraging exploration in off-policy algorithms via normalizing flowscited by 46
- 2020The Neural Coding Framework for Learning Generative Modelscited by 44
- 2022Transformers are Meta-Reinforcement Learnerscited by 41
- 2014Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPscited by 28
- 2020Improving Robustness via Risk Averse Distributional Reinforcement Learningcited by 28
- 2019Normalizing Flows: An Introduction and Review of Current Methodscited by 16
- 2020Implicit Distributional Reinforcement Learningcited by 16
- 2021GMAC: A Distributional Perspective on Actor-Critic Frameworkcited by 16
- 2021Risk-Conditioned Distributional Soft Actor-Critic for Risk-Sensitive Navigationcited by 15
- 2020Sample-based Distributional Policy Gradientcited by 14
- 2021Controlling Wasserstein Distances by Kernel Norms with Application to Compressive Statistical Learningcited by 14
- 2020Distributional Reinforcement Learning via Moment Matchingcited by 10
- 2018A transport-based multifidelity preconditioner for Markov chain Monte Carlocited by 9
- 2022Distributional Reinforcement Learning with Regularized Wasserstein Losscited by 8
- 2021Conjugated Discrete Distributions for Distributional Reinforcement Learningcited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.