Scaling Distributed Multi-task Reinforcement Learning with Experience Sharing
Not on the map. Why?
Abstract (arXiv)
Recently, DARPA launched the ShELL program, which aims to explore how experience sharing can benefit distributed lifelong learning agents in adapting to new challenges. In this paper, we address this issue by conducting both theoretical and empirical research on distributed multi-task reinforcement learning (RL), where a group of N agents collaboratively solves M tasks without prior knowledge of their identities. We approach the problem by formulating it as linearly parameterized contextual Markov decision processes (MDPs), where each task is represented by a context that specifies the transition dynamics and rewards. To tackle this problem, we propose an algorithm called DistMT-LSVI. First, the agents identify the tasks, and then they exchange information through a central server to derive ε-optimal policies for the tasks. Our research demonstrates that to achieve ε-optimal policies for all M tasks, a single agent using DistMT-LSVI needs to run a total number of episodes that is at most tildeO(d3H6(ε-2+crm sep-2)· M/N), where crm sep>0 is a constant representing task separability, H is the horizon of each episode, and d is the feature dimension of the dynamics and rewards. Notably, DistMT-LSVI improves the sample complexity of non-distributed settings by a factor of 1/N, as each agent independently learns ε-optimal policies for all M tasks using tildeO(d3H6Mε-2) episodes. Additionally, we provide numerical experiments conducted on OpenAI Gym Atari environments that validate our theoretical findings.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 11 Jul 2023.
Builds on
24 of 41 references.
References · 24
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
- 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
- 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
- 2015Contextual Markov Decision Processescited by 220
- 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 194
- 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 174
- 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 143
- 2018Multi-task Deep Reinforcement Learning with PopArtcited by 141
- 2018Dynamic Weights in Multi-Objective Deep Reinforcement Learningcited by 100
- 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 90
- 2020On Reward-Free Reinforcement Learning with Linear Function Approximationcited by 89
12 more
- 2013Sample Complexity of Multi-task Reinforcement Learningcited by 58
- 2017Markov Decision Processes with Continuous Side Informationcited by 56
- 2021From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligencecited by 35
- 2021Lifelong Robotic Reinforcement Learning by Retaining Experiencescited by 34
- 2020Lipschitz Lifelong Reinforcement Learningcited by 23
- 2021Provably Efficient Cooperative Multi-Agent Reinforcement Learning with Function Approximationcited by 23
- 2021Provably Efficient Multi-Task Reinforcement Learning with Model Transfercited by 16
- 2020Accommodating Picky Customers: Regret Bound and Exploration Complexity for Multi-Objective Reinforcement Learningcited by 11
- 2022Learning Efficiently Function Approximation for Contextual MDPcited by 10
- 2014Online learning in MDPs with side informationcited by 7
- 2020TempLe: Learning Template of Transitions for Sample Efficient Multi-task RLcited by 6
- 2019Continuous Control with Contexts, Provablycited by 3
Built on by
Citing papers per year: 2025: 1.
1 paper.