Oracle-Efficient Reinforcement Learning for Max Value Ensembles
Not on the map. Why?
Abstract (arXiv)
Reinforcement learning (RL) in large or infinite state spaces is notoriously challenging, both theoretically (where worst-case sample and computational complexities must scale with state space cardinality) and experimentally (where function approximation and policy gradient techniques often scale poorly and suffer from instability and high variance). One line of research attempting to address these difficulties makes the natural assumption that we are given a collection of heuristic base or textitconstituent policies upon which we would like to improve in a scalable manner. In this work we aim to compete with the textitmax-following policy, which at each state follows the action of whichever constituent policy has the highest value. The max-following policy is always at least as good as the best constituent policy, and may be considerably better. Our main result is an efficient algorithm that learns to compete with the max-following policy, given only access to the constituent policies (but not their value functions). In contrast to prior work in similar settings, our theoretical results require only the minimal assumption of an ERM oracle for value function approximation for the constituent policies (and not the global optimal policy or the max-following policy itself) on samplable distributions. We illustrate our algorithm's experimental effectiveness and behavior on several robotic simulation testbeds.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 27 of its 42 references that are in the map. First posted on arXiv 27 May 2024.
Builds on
27 of 42 references.
References · 27
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
- 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
- 2020dm_control: Software and Tasks for Continuous Controlcited by 368
- 2016Successor Features for Transfer in Reinforcement Learningcited by 331
- 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
- 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
- 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
- 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
- 2020Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcited by 104
- 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
15 more
- 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
- 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 65
- 2015Learning to Search Better Than Your Teachercited by 50
- 2020Discount Factor as a Regularizer in Reinforcement Learningcited by 44
- 2021Ensemble Bootstrapping for Q-Learningcited by 30
- 2019AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teacherscited by 27
- 2022CompoSuite: A Compositional Reinforcement Learning Benchmarkcited by 21
- 2023Ensemble Reinforcement Learning: A Surveycited by 16
- 2023Blending Imitation and Reinforcement Learning for Robust Policy Improvementcited by 16
- 2021A Boosting Approach to Reinforcement Learningcited by 12
- 2018OIL: Observational Imitation Learningcited by 11
- 2023Active Policy Improvement from Multiple Black-box Oraclescited by 11
- 2024Exploration is Harder than Prediction: Cryptographically Separating Reinforcement Learning from Supervised Learningcited by 6
- 2023Robotic Manipulation Datasets for Offline Compositional Reinforcement Learningcited by 5
- 2022Actor-Critic based Improper Reinforcement Learningcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.