mlsanityData to 7 Oct 2026

Task-Agnostic Continual Reinforcement Learning: Gaining Insights and Overcoming Challenges

arXiv 2205.14495 ↗First posted 28 May 2022cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Continual learning (CL) enables the development of models and agents that learn from a sequence of tasks while addressing the limitations of standard deep learning approaches, such as catastrophic forgetting. In this work, we investigate the factors that contribute to the performance differences between task-agnostic CL and multi-task (MTL) agents. We pose two hypotheses: (1) task-agnostic methods might provide advantages in settings with limited data, computation, or high dimensionality, and (2) faster adaptation may be particularly beneficial in continual learning settings, helping to mitigate the effects of catastrophic forgetting. To investigate these hypotheses, we introduce a replay-based recurrent reinforcement learning (3RL) methodology for task-agnostic CL agents. We assess 3RL on a synthetic task and the Meta-World benchmark, which includes 50 unique manipulation tasks. Our results demonstrate that 3RL outperforms baseline methods and can even surpass its multi-task equivalent in challenging settings with high dimensionality. We also show that the recurrent task-agnostic agent consistently outperforms or matches the performance of its transformer-based counterpart. These findings provide insights into the advantages of task-agnostic CL over task-aware MTL approaches and highlight the potential of task-agnostic methods in resource-constrained, high-dimensional, and multi-task environments.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 54 of its 87 references that are in the map. First posted on arXiv 28 May 2022.

Builds on

54 of 87 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  10. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  12. 2018Experience Replay for Continual Learningcited by 1,074
38 more
  1. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  2. 2017A Survey on Multi-Task Learningcited by 838
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  4. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  6. 2019Three scenarios for continual learningcited by 610
  7. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  8. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
  9. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  10. 2019On Tiny Episodic Memories in Continual Learningcited by 299
  11. 2019Challenges of Real-World Reinforcement Learningcited by 282
  12. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  13. 2018Selective Experience Replay for Lifelong Learningcited by 239
  14. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  15. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  16. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
  17. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 112
  18. 2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 80
  19. 2015Memory-based control with recurrent neural networkscited by 77
  20. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  21. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 68
  22. 2019Meta-Q-Learningcited by 62
  23. 2018Task Agnostic Continual Learning Using Online Variational Bayescited by 61
  24. 2019Task Agnostic Continual Learning via Meta Learningcited by 53
  25. 2020Deep Reinforcement Learning amidst Lifelong Non-Stationaritycited by 42
  26. 2021Same State, Different Task: Continual Reinforcement Learning without Interferencecited by 42
  27. 2021Understanding Continual Learning Settings with Data Distribution Drift Analysiscited by 38
  28. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
  29. 2019DisCoRL: Continual Reinforcement Learning via Policy Distillationcited by 35
  30. 2021Continual Learning for Recurrent Neural Networks: an Empirical Evaluationcited by 34
  31. 2019Regularization Shortcomings for Continual Learningcited by 32
  32. 2018Continual Reinforcement Learning with Complex Synapsescited by 30
  33. 2020Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgettingcited by 29
  34. 2020Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processescited by 23
  35. 2021Sequoia: A Software Framework to Unify Continual Learning Researchcited by 19
  36. 2020Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processescited by 17
  37. 2021CoMPS: Continual Meta Policy Searchcited by 15
  38. 2019Continual Learning of New Sound Classes using Generative Replaycited by 11

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 3; 2025: 2; 2026: 3.

10 papers.

  1. 2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Modelscited by 48
  2. 2025A Survey of Continual Reinforcement Learningcited by 16
  3. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Datacited by 13
  4. 2024Multi-granularity Knowledge Transfer for Continual Reinforcement Learningcited by 6
  5. 2024Statistical Context Detection for Deep Lifelong Reinforcement Learningcited by 6
  6. 2025Action-Adaptive Continual Learning: Enabling Policy Generalization under Dynamic Action Spacescited by 1
  7. 2026Continual Policy Consolidation for Lifelong Robot Learningcited by 1
  8. 2022Task-Agnostic Learning to Accomplish New Taskscited by 0
  9. 2026Task diversity produces systematic transfer but inhibits continual reinforcement learningcited by 0
  10. 2026Space-sampled Value Decay: Forgetting Mechanisms for Non-stationary Reinforcement Learningcited by 0