mlsanityData to 7 Oct 2026

CoMPS: Continual Meta Policy Search

arXiv 2112.04467 ↗First posted 8 Dec 2021cs.LG cs.AI cs.ROcited by 15

Not on the map. Why?

Abstract (arXiv)

We develop a new continual meta-learning method to address challenges in sequential multi-task learning. In this setting, the agent's goal is to achieve high reward over any sequence of tasks quickly. Prior meta-reinforcement learning algorithms have demonstrated promising results in accelerating the acquisition of new tasks. However, they require access to all tasks during training. Beyond simply transferring past experience to new tasks, our goal is to devise continual reinforcement learning algorithms that learn to learn, using their experience on previous tasks to learn new tasks more quickly. We introduce a new method, continual meta-policy search (CoMPS), that removes this limitation by meta-training in an incremental fashion, over each task in a sequence, without revisiting prior tasks. CoMPS continuously repeats two subroutines: learning a new task using RL and using the experience from RL to perform completely offline meta-learning to prepare for subsequent task learning. We find that CoMPS outperforms prior continual learning and off-policy meta-reinforcement methods on several sequences of challenging continuous control tasks.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 57 of its 74 references that are in the map. First posted on arXiv 8 Dec 2021.

Builds on

57 of 74 references.

References · 57

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2016Progressive Neural Networkscited by 1,332
  5. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2018Experience Replay for Continual Learningcited by 1,074
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  9. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2019Introduction to Online Convex Optimizationcited by 793
  12. 2017Deep Reinforcement Learning that Matterscited by 727
38 more
  1. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  2. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  3. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  4. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  5. 2015Policy Distillationcited by 414
  6. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  7. 2016Learning to reinforcement learncited by 369
  8. 2017Variational Continual Learningcited by 335
  9. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  10. 2018Meta-Learning: A Surveycited by 254
  11. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  12. 2018Selective Experience Replay for Lifelong Learningcited by 239
  13. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  14. 2018Task-Free Continual Learningcited by 219
  15. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  16. 2019Meta-Learning Representations for Continual Learningcited by 189
  17. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  18. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
  19. 2019DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehensioncited by 156
  20. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  21. 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 128
  22. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 106
  23. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  24. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  25. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  26. 2018Evolved Policy Gradientscited by 86
  27. 2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 80
  28. 2017Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithmcited by 65
  29. 2019Meta-Learning via Learned Losscited by 60
  30. 2019Task Agnostic Continual Learning via Meta Learningcited by 53
  31. 2020La-MAML: Look-ahead Meta Learning for Continual Learningcited by 51
  32. 2019Continuous Meta-Learning without Taskscited by 50
  33. 2016Learning to Perform Physics Experiments via Deep Reinforcement Learningcited by 47
  34. 2018Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgettingcited by 46
  35. 2018Meta Reinforcement Learning with Latent Variable Gaussian Processescited by 44
  36. 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 43
  37. 2020Meta-Consolidation for Continual Learningcited by 40
  38. 2018Unsupervised Meta-Learning for Reinforcement Learningcited by 38

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 4; 2024: 1; 2025: 4; 2026: 2.

15 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  3. 2022Building a Subspace of Policies for Scalable Continual Learningcited by 28
  4. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  5. 2025A Survey of Continual Reinforcement Learningcited by 16
  6. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 13
  7. 2023Continual Task Allocation in Meta-Policy Network via Sparse Promptingcited by 11
  8. 2022Task-Agnostic Continual Reinforcement Learning: Gaining Insights and Overcoming Challengescited by 10
  9. 2023Mnemosyne: Learning to Train Transformers with Transformerscited by 9
  10. 2024Multi-granularity Knowledge Transfer for Continual Reinforcement Learningcited by 6
  11. 2022Continual Meta-Reinforcement Learning for UAV-Aided Vehicular Wireless Networkscited by 3
  12. 2025Overcoming Non-stationary Dynamics with Evidential Proximal Policy Optimizationcited by 2
3 more
  1. 2023Meta-World Conditional Neural Processescited by 1
  2. 2025Meta-Reinforcement Learning with Discrete World Models for Adaptive Load Balancingcited by 0
  3. 2026Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learningcited by 0