Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimization
Not on the map. Why?
Abstract (arXiv)
Recent success in Deep Reinforcement Learning (DRL) methods has shown that policy optimization with respect to an off-policy distribution via importance sampling is effective for sample reuse. In this paper, we show that the use of importance sampling could introduce high variance in the objective estimate. Specifically, we show in a principled way that the variance of importance sampling estimate grows quadratically with importance ratios and the large ratios could consequently jeopardize the effectiveness of surrogate objective optimization. We then propose a technique called sample dropout to bound the estimation variance by dropping out samples when their ratio deviation is too high. We instantiate this sample dropout technique on representative policy optimization algorithms, including TRPO, PPO, and ESPO, and demonstrate that it consistently boosts the performance of those DRL algorithms on both continuous and discrete action controls, including MuJoCo, DMControl and Atari video games. Our code is open-sourced at urlhttps://github.com/LinZichuan/sdpo.git.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 28 of its 48 references that are in the map. First posted on arXiv 5 Feb 2023.
Builds on
28 of 48 references.
References · 28
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2015Trust Region Policy Optimizationcited by 2,503
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
- 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
- 2015Prioritized Experience Replaycited by 1,070
- 2018DeepMind Control Suitecited by 935
- 2017Constrained Policy Optimizationcited by 928
- 2015End-to-End Training of Deep Visuomotor Policiescited by 847
- 2017Deep Reinforcement Learning that Matterscited by 727
- 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
- 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
16 more
- 2018Self-Imitation Learningcited by 135
- 2020Mirror Descent Policy Optimizationcited by 102
- 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 84
- 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 77
- 2019Truly Proximal Policy Optimizationcited by 68
- 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 65
- 2018Policy Optimization via Importance Samplingcited by 47
- 2018Episodic Memory Deep Q-Networkscited by 41
- 2021Generalized Proximal Policy Optimization with Sample Reusecited by 39
- 2021JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learningcited by 28
- 2020Guided Exploration with Proximal Policy Optimization using a Single Demonstrationcited by 19
- 2022You May Not Need Ratio Clipping in PPOcited by 19
- 2018A Closer Look at Deep Policy Gradientscited by 12
- 2021PTR-PPO: Proximal Policy Optimization with Prioritized Trajectory Replaycited by 9
- 2019Combining Experience Replay with Exploration by Random Network Distillationcited by 5
- 2020Continuous-action Reinforcement Learning for Playing Racing Games: Comparing SPG to PPOcited by 4
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 1.
3 papers.