On the Convergence Theory of Meta Reinforcement Learning with Personalized Policies
Not on the map. Why?
Abstract (arXiv)
Modern meta-reinforcement learning (Meta-RL) methods are mainly developed based on model-agnostic meta-learning, which performs policy gradient steps across tasks to maximize policy performance. However, the gradient conflict problem is still poorly understood in Meta-RL, which may lead to performance degradation when encountering distinct tasks. To tackle this challenge, this paper proposes a novel personalized Meta-RL (pMeta-RL) algorithm, which aggregates task-specific personalized policies to update a meta-policy used for all tasks, while maintaining personalized policies to maximize the average return of each task under the constraint of the meta-policy. We also provide the theoretical analysis under the tabular setting, which demonstrates the convergence of our pMeta-RL algorithm. Moreover, we extend the proposed pMeta-RL algorithm to a deep network version based on soft actor-critic, making it suitable for continuous control tasks. Experiment results show that the proposed algorithms outperform other previous Meta-RL algorithms on Gym and MuJoCo suites.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 40 references that are in the map. First posted on arXiv 21 Sep 2022.
Builds on
31 of 40 references.
References · 31
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
- 2016OpenAI Gymcited by 1,684
- 2020Gradient Surgery for Multi-Task Learningcited by 1,210
- 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
- 2015End-to-End Training of Deep Visuomotor Policiescited by 847
- 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
- 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
- 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
- 2017Distral: Robust Multitask Reinforcement Learningcited by 231
19 more
- 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
- 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
- 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
- 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
- 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
- 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
- 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
- 2019Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulationcited by 103
- 2018ProMP: Proximal Meta-Policy Searchcited by 91
- 2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 77
- 2019Efficient Ridesharing Order Dispatching with Mean Field Multi-Agent Reinforcement Learningcited by 57
- 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 52
- 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 43
- 2016The Curious Robot: Learning Visual Representations via Physical Interactionscited by 37
- 2016Learning to Push by Grasping: Using multiple tasks for effective learningcited by 36
- 2020Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Controlcited by 27
- 2020MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Explorationcited by 24
- 2021HetMAML: Task-Heterogeneous Model-Agnostic Meta-Learning for Few-Shot Learning Across Modalitiescited by 8
- 2019MAME : Model-Agnostic Meta-Explorationcited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.