Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective
Not on the map. Why?
Abstract (arXiv)
Off-policy Actor-Critic algorithms have demonstrated phenomenal experimental performance but still require better explanations. To this end, we show its policy evaluation error on the distribution of transitions decomposes into: a Bellman error, a bias from policy mismatch, and a variance term from sampling. By comparing the magnitude of bias and variance, we explain the success of the Emphasizing Recent Experience sampling and 1/age weighted sampling. Both sampling strategies yield smaller bias and variance and are hence preferable to uniform sampling.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 6 Oct 2021.
Builds on
33 of 43 references.
References · 33
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2015Trust Region Policy Optimizationcited by 2,580
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2017Improved Training of Wasserstein GANscited by 2,235
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
- 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
- 2015Prioritized Experience Replaycited by 1,099
- 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
- 2018Is Q-learning Provably Efficient?cited by 487
- 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
- 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
21 more
- 2018Distributed Prioritized Experience Replaycited by 269
- 2019Batch Policy Learning under Constraintscited by 244
- 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
- 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 140
- 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
- 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 119
- 2020Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reductioncited by 95
- 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
- 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
- 2018Remember and Forget for Experience Replaycited by 43
- 2020Experience Replay with Likelihood-free Importance Weightscited by 42
- 2019Off-Policy Actor-Critic with Shared Experience Replaycited by 39
- 2019Learning to Combat Compounding-Error in Model-Based Reinforcement Learningcited by 39
- 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
- 2020Near-Optimal Provable Uniform Convergence in Offline Policy Evaluation for Reinforcement Learningcited by 26
- 2018Clipped Action Policy Gradientcited by 19
- 2019Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Pastcited by 18
- 2020Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policiescited by 10
- 2019Learning to Control in Metric Space with Optimal Regretcited by 9
- 2020A maximum-entropy approach to off-policy evaluation in average-reward MDPscited by 9
- 2020A Contraction Approach to Model-based Reinforcement Learningcited by 1
Built on by
Citing papers per year: 2022: 1; 2023: 1.
2 papers.