mlsanityData to 14 Oct 2026

Modular Lifelong Reinforcement Learning via Neural Composition

arXiv 2207.00429 ↗First posted 1 Jul 2022cs.LG cs.AIcited by 40

Not on the map. Why?

Abstract (arXiv)

Humans commonly solve complex problems by decomposing them into easier subproblems and then combining the subproblem solutions. This type of compositional reasoning permits reuse of the subproblem solutions when tackling future tasks that share part of the underlying compositional structure. In a continual or lifelong reinforcement learning (RL) setting, this ability to decompose knowledge into reusable components would enable agents to quickly learn new RL tasks by leveraging accumulated compositional structures. We explore a particular form of composition based on neural modules and present a set of RL problems that intuitively admit compositional solutions. Empirically, we demonstrate that neural composition indeed captures the underlying structure of this space of problems. We further propose a compositional lifelong RL method that leverages accumulated neural components to accelerate the learning of future tasks while retaining performance on previous tasks via off-line RL over replayed experiences.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 43 of its 58 references that are in the map. First posted on arXiv 1 Jul 2022.

Builds on

43 of 58 references.

References · 43

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2016Learning without Forgettingcited by 2,496
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2018Experience Replay for Continual Learningcited by 1,095
  7. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  9. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  10. 2018Learning Dexterous In-Hand Manipulationcited by 687
  11. 2019Gradient based sample selection for online continual learningcited by 571
  12. 2018Progress & Compress: A scalable framework for continual learningcited by 520
31 more
  1. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  2. 2016The Option-Critic Architecturecited by 448
  3. 2015Neural Module Networkscited by 376
  4. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 364
  5. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  6. 2017Variational Continual Learningcited by 339
  7. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  8. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  9. 2018Selective Experience Replay for Lifelong Learningcited by 241
  10. 2019Recurrent Independent Mechanismscited by 219
  11. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 201
  12. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  13. 2019Continual Unsupervised Representation Learningcited by 147
  14. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  15. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 143
  16. 2016Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfercited by 127
  17. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 126
  18. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 111
  19. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  20. 2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 72
  21. 2018Modular Networks: Learning to Decompose Neural Computationcited by 62
  22. 2018Modular meta-learningcited by 51
  23. 2020A Boolean Task Algebra for Reinforcement Learningcited by 48
  24. 2018Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologiescited by 43
  25. 2018HOUDINI: Lifelong Learning as Program Synthesiscited by 38
  26. 2020Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modulescited by 38
  27. 2018Automatically Composing Representation Transformations as a Means for Generalizationcited by 35
  28. 2020Lifelong Learning of Compositional Structurescited by 35
  29. 2020Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgettingcited by 29
  30. 2017Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Orderingcited by 28
  31. 2020Evaluating Logical Generalization in Graph Neural Networkscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 15; 2024: 7; 2025: 12; 2026: 2.

40 papers.

Sort:

By area:

  1. 2023LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discoverycited by 67
  2. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  3. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  4. 2022Lifelong Reinforcement Learning with Modulating Maskscited by 22
  5. 2022CompoSuite: A Compositional Reinforcement Learning Benchmarkcited by 21
  6. 2025A Survey of Continual Reinforcement Learningcited by 18
  7. 2023Embodied Lifelong Learning for Task and Motion Planningcited by 17
  8. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  9. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 12
  10. 2022Beyond Supervised Continual Learning: a Reviewcited by 11
  11. 2025Self-Composing Policies for Scalable Continual Reinforcement Learningcited by 11
  12. 2023Sharing Lifelong Reinforcement Learning Knowledge via Modulating Maskscited by 10
28 more
  1. 2023Task Generalization with Stability Guarantees via Elastic Dynamical System Motion Policiescited by 10
  2. 2023Elephant Neural Networks: Born to Be a Continual Learnercited by 10
  3. 2023Decision Stacks: Flexible Reinforcement Learning via Modular Generative Modelscited by 8
  4. 2023Egocentric Planning for Scalable Embodied Task Achievementcited by 7
  5. 2023Robotic Manipulation Datasets for Offline Compositional Reinforcement Learningcited by 6
  6. 2023Evaluating Continual Learning on a Home Robotcited by 5
  7. 2025Continual Knowledge Adaptation for Reinforcement Learningcited by 5
  8. 2023Visually Grounded Continual Language Learning with Selective Specializationcited by 4
  9. 2024Reset & Distill: A Recipe for Overcoming Negative Transfer in Continual Reinforcement Learningcited by 4
  10. 2023Continual Visual Reinforcement Learning with A Life-Long World Modelcited by 3
  11. 2024R3L: Relative Representations for Reinforcement Learningcited by 2
  12. 2024Reduce, Reuse, Recycle: Categories for Compositional Reinforcement Learningcited by 2
  13. 2024The State of Robot Motion Generationcited by 2
  14. 2023Towards Zero-Shot Functional Compositionality of Language Modelscited by 1
  15. 2023Heterogeneous Knowledge for Augmented Modular Reinforcement Learningcited by 1
  16. 2023Bridging Neural and Symbolic Representations with Transitional Dictionary Learningcited by 1
  17. 2024Continual Model-based Reinforcement Learning for Data Efficient Wireless Network Optimisationcited by 1
  18. 2025State Combinatorial Generalization In Decision Making With Conditional Diffusion Modelscited by 1
  19. 2025Policy Search, Retrieval, and Composition via Task Similarity in Collaborative Agentic Systemscited by 1
  20. 2024Unveiling Options with Neural Decompositioncited by 0
  21. 2025Mapping representations in Reinforcement Learning via Semantic Alignment for Zero-Shot Stitchingcited by 0
  22. 2025Neuron-level Balance between Stability and Plasticity in Deep Reinforcement Learningcited by 0
  23. 2025What Neuroscience Can Teach AI About Learning in Continuously Changing Environmentscited by 0
  24. 2025Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functionscited by 0
  25. 2025Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machinescited by 0
  26. 2025Iterative Compositional Data Generation for Robot Controlcited by 0
  27. 2026Learning Transferable Skills in Action RPGs via Directed Skill Graphs and Selective Adaptationcited by 0
  28. 2026How to Find and Reuse Policies for Continuous Adaptation in Lifelong Reinforcement Learningcited by 0