Distillation Policy Optimization
Not on the map. Why?
Abstract (arXiv)
While on-policy algorithms are known for their stability, they often demand a substantial number of samples. In contrast, off-policy algorithms, which leverage past experiences, are considered sample-efficient but tend to exhibit instability. Can we develop an algorithm that harnesses the benefits of off-policy data while maintaining stable learning? In this paper, we introduce an actor-critic learning framework that harmonizes two data sources for both evaluation and control, facilitating rapid learning and adaptable integration with on-policy algorithms. This framework incorporates variance reduction mechanisms, including a unified advantage estimator (UAE) and a residual baseline, improving the efficacy of both on- and off-policy learning. Our empirical results showcase substantial enhancements in sample efficiency for on-policy algorithms, effectively bridging the gap to the off-policy approaches. It demonstrates the promise of our approach as a novel learning paradigm.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 1 Feb 2023.
Builds on
32 of 52 references.
References · 32
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2015Trust Region Policy Optimizationcited by 2,503
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
- 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
- 2017A Distributional Perspective on Reinforcement Learningcited by 743
20 more
- 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
- 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
- 2019A Theoretical Analysis of Deep Q-Learningcited by 212
- 2018Distributed Distributional Deterministic Policy Gradientscited by 195
- 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
- 2014Neural Variational Inference and Learning in Belief Networkscited by 116
- 2017Backpropagation through the Void: Optimizing control variates for black-box gradient estimationcited by 104
- 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 84
- 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 77
- 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 65
- 2018The Mirage of Action-Dependent Baselines in Reinforcement Learningcited by 46
- 2017Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learningcited by 40
- 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
- 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 31
- 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30
- 2016PLATO: Policy Learning using Adaptive Trajectory Optimizationcited by 23
- 2016Combining policy gradient and Q-learningcited by 23
- 2019Distributional Policy Optimization: An Alternative Approach for Continuous Controlcited by 21
- 2018Clipped Action Policy Gradientcited by 17
- 2022Revisiting Gaussian mixture critics in off-policy reinforcement learning: a sample-based approachcited by 6
Built on by
Citing papers per year: 2023: 1.
1 paper.
- 2023Discerning Temporal Difference Learningcited by 0
- 2023Discerning Temporal Difference Learning12 Oct 2023