Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Tasks
Not on the map. Why?
Abstract (arXiv)
Multitask Reinforcement Learning (MTRL) approaches have gained increasing attention for its wide applications in many important Reinforcement Learning (RL) tasks. However, while recent advancements in MTRL theory have focused on the improved statistical efficiency by assuming a shared structure across tasks, exploration--a crucial aspect of RL--has been largely overlooked. This paper addresses this gap by showing that when an agent is trained on a sufficiently diverse set of tasks, a generic policy-sharing algorithm with myopic exploration design like ε-greedy that are inefficient in general can be sample-efficient for MTRL. To the best of our knowledge, this is the first theoretical demonstration of the "exploration benefits" of MTRL. It may also shed light on the enigmatic success of the wide applications of myopic exploration in practice. To validate the role of diversity, we conduct experiments on synthetic robotic control environments, where the diverse task set aligns with the task selection by automatic curriculum learning, which is empirically shown to improve sample-efficiency.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 51 of its 67 references that are in the map. First posted on arXiv 3 Mar 2024.
Builds on
51 of 67 references.
References · 51
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2017Hindsight Experience Replaycited by 1,096
- 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
- 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
- 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
- 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
- 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
- 2013Learning to Optimize Via Posterior Samplingcited by 253
- 2017Automated Curriculum Learning for Neural Networkscited by 251
- 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
- 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
- 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
38 more
- 2015The Benefit of Multitask Representation Learningcited by 194
- 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
- 2020Reward-Free Exploration for Reinforcement Learningcited by 177
- 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
- 2017Deep Exploration via Randomized Value Functionscited by 155
- 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
- 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
- 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
- 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
- 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
- 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 116
- 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
- 2019Optimism in Reinforcement Learning with Generalized Linear Function Approximationcited by 111
- 2020Naive Exploration is Optimal for Online LQRcited by 111
- 2020Is Reinforcement Learning More Difficult Than Bandits? A Near-optimal Algorithm Escaping the Curse of Horizoncited by 99
- 2020On Reward-Free Reinforcement Learning with Linear Function Approximationcited by 89
- 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
- 2019Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activitycited by 70
- 2014Curriculum Learning of Multiple Taskscited by 65
- 2013Sample Complexity of Multi-task Reinforcement Learningcited by 58
- 2016A Reinforcement Learning System to Encourage Physical Activity in Diabetes Patientscited by 38
- 2022Provable Benefits of Representational Transfer in Reinforcement Learningcited by 38
- 2023Robotic Offline RL from Internet Videos via Value-Function Pre-Trainingcited by 31
- 2021On the Power of Multitask Representation Learning in Linear MDPcited by 30
- 2020Temporally-Extended ε-Greedy Explorationcited by 29
- 2022Provable Benefit of Multitask Representation Learning in Reinforcement Learningcited by 29
- 2022Guarantees for Epsilon-Greedy Reinforcement Learning with Function Approximationcited by 29
- 2022On the Statistical Efficiency of Reward-Free Exploration in Non-Linear RLcited by 27
- 2021Representation Learning Beyond Linear Prediction Functionscited by 21
- 2021TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RLcited by 18
- 2022Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemblecited by 18
- 2021Provably Efficient Multi-Task Reinforcement Learning with Model Transfercited by 16
- 2022Nearly Minimax Algorithms for Linear Bandits with Shared Representationcited by 16
- 2022Understanding the Complexity Gains of Single-Task RL with a Curriculumcited by 16
- 2021Improved No-Regret Algorithms for Stochastic Shortest Path with Linear MDPcited by 14
- 2018When Simple Exploration is Sample Efficient: Identifying Sufficient Conditions for Random Exploration to Yield PAC RL Algorithmscited by 11
- 2021On the Statistical Benefits of Curriculum Learningcited by 10
- 2023Did we personalize? Assessing personalization by an online reinforcement learning algorithm using resamplingcited by 6
50 most cited of 51 All 51 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.