# Continual Policy Consolidation for Lifelong Robot Learning

arXiv: https://arxiv.org/abs/2601.22475 · HTML: https://mlsanity.com/paper/2601.22475/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 34 of its 38 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 34)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [Multi-Task Reinforcement Learning with Context-based Representations](https://mlsanity.com/paper/2102.06177/) (2021)
- [Multi-Task Reinforcement Learning with Soft Modularization](https://mlsanity.com/paper/2003.13661/) (2020)
- [Continual World: A Robotic Benchmark For Continual Reinforcement Learning](https://mlsanity.com/paper/2105.10919/) (2021)
- [Lifelong Language Pretraining with Distribution-Specialized Experts](https://mlsanity.com/paper/2305.12281/) (2023)
- [Theory on Mixture-of-Experts in Continual Learning](https://mlsanity.com/paper/2406.16437/) (2024)
- [Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts](https://mlsanity.com/paper/2311.11385/) (2023)
- [PaCo: Parameter-Compositional Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2210.11653/) (2022)
- [Maintaining Plasticity in Deep Continual Learning](https://mlsanity.com/paper/2306.13812/) (2023)
- [Meta-World+: An Improved, Standardized, RL Benchmark](https://mlsanity.com/paper/2505.11289/) (2025)
- [Policy Consolidation for Continual Reinforcement Learning](https://mlsanity.com/paper/1902.00255/) (2019)
- [DisCoRL: Continual Reinforcement Learning via Policy Distillation](https://mlsanity.com/paper/1907.05855/) (2019)
- [Disentangling Transfer in Continual Reinforcement Learning](https://mlsanity.com/paper/2209.13900/) (2022)
- [Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning](https://mlsanity.com/paper/2405.16642/) (2024)
- [Task-Agnostic Continual Reinforcement Learning: Gaining Insights and Overcoming Challenges](https://mlsanity.com/paper/2205.14495/) (2022)
- [Continual Task Allocation in Meta-Policy Network via Sparse Prompting](https://mlsanity.com/paper/2305.18444/) (2023)
- [Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learning](https://mlsanity.com/paper/2506.05985/) (2025)

## Built on by (most cited first, 3 of 3)

- [CIDER: Continual Interactive Distillation for Embodied Reinforcement Learning](https://mlsanity.com/paper/2608.21899/) (2026)
- [Vision Transformers that Never Stop Learning](https://mlsanity.com/paper/2603.07787/) (2026)
- [Dream to Remember: Continual Steering of Generalist Robot Policies with World Models](https://mlsanity.com/paper/2610.13816/) (2026)

## Abstract (arXiv)

Building a generalist robot policy requires continuously integrating new skills while preserving previously acquired behaviors. Directly optimizing a single policy over a growing task stream is difficult because robotic interaction is expensive, task distributions are heterogeneous, and sequential updates induce interference. To address these problems, we propose continual policy consolidation (CPC), a teacher--student framework that combines continual policy distillation with prioritized experience replay and expandable experts. This architecture separates skill acquisition from policy consolidation: teachers are trained independently through reinforcement learning, and their behaviors are continually distilled into a central generalist student. This decomposition retains the practical strength of reinforcement learning for task-specialized training while casting student-side consolidation as a supervised policy-learning problem. To balance stability and plasticity as the task stream grows, the student combines an expandable Transformer-based mixture-of-experts architecture with prioritized trajectory replay. Extensive experiments show that the student recovers a large proportion of teacher performance while achieving near-zero forgetting. These results demonstrate a scalable route for consolidating independently acquired robot skills into a continually growing generalist policy.