# M2Distill: Multi-Modal Distillation for Lifelong Imitation Learning

arXiv: https://arxiv.org/abs/2410.00064 · HTML: https://mlsanity.com/paper/2410.00064/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion](https://mlsanity.com/paper/2111.11326/) (2021)
- [Incremental Learning of Object Detectors without Catastrophic Forgetting](https://mlsanity.com/paper/1708.06977/) (2017)
- [Uncertainty-based Continual Learning with Adaptive Regularization](https://mlsanity.com/paper/1905.11614/) (2019)
- [Language-Conditioned Imitation Learning for Robot Manipulation Tasks](https://mlsanity.com/paper/2010.12083/) (2020)
- [Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation](https://mlsanity.com/paper/2109.13841/) (2021)
- [Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation](https://mlsanity.com/paper/2307.03659/) (2023)
- [Generative Models from the perspective of Continual Learning](https://mlsanity.com/paper/1812.09111/) (2018)
- [LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discovery](https://mlsanity.com/paper/2311.02058/) (2023)
- [CRIL: Continual Robot Imitation Learning via Generative and Prediction Model](https://mlsanity.com/paper/2106.09422/) (2021)
- [Subspace Distillation for Continual Learning](https://mlsanity.com/paper/2307.16419/) (2023)
- [PolyTask: Learning Unified Policies through Behavior Distillation](https://mlsanity.com/paper/2310.08573/) (2023)
- [L3DMC: Lifelong Learning using Distillation via Mixed-Curvature Space](https://mlsanity.com/paper/2307.16459/) (2023)

## Built on by (most cited first, 11 of 11)

- [Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trends](https://mlsanity.com/paper/2506.20966/) (2025)
- [Continual Learning for Generative AI: From LLMs to MLLMs and Beyond](https://mlsanity.com/paper/2506.13045/) (2025)
- [SPECI: Skill Prompts based Hierarchical Continual Imitation Learning for Robot Manipulation](https://mlsanity.com/paper/2504.15561/) (2025)
- [Few-Shot Vision-Language Action-Incremental Policy Learning](https://mlsanity.com/paper/2504.15517/) (2025)
- [T2S: Tokenized Skill Scaling for Lifelong Imitation Learning](https://mlsanity.com/paper/2508.01167/) (2025)
- [SPREAD: Subspace Representation Distillation for Lifelong Imitation Learning](https://mlsanity.com/paper/2603.08763/) (2026)
- [Lifelong Imitation Learning with Multimodal Latent Replay and Incremental Adjustment](https://mlsanity.com/paper/2603.10929/) (2026)
- [Learning New Tasks via Reusable Skills: Skill-Compositional Experts for Embodied Continual Learning](https://mlsanity.com/paper/2606.15685/) (2026)
- [LiMoDE: Rethinking Lifelong Robot Manipulation from a Mixture-of-Dynamic-Experts Perspective](https://mlsanity.com/paper/2606.26183/) (2026)
- [Federated Subspace Guided Vision-Language-Action Policy Distillation for Non-IID Multi-Robot Manipulation](https://mlsanity.com/paper/2609.32239/) (2026)
- [Data-Free Continual Learning for World Action Model with Rectified Generative Replay](https://mlsanity.com/paper/2610.18038/) (2026)

## Abstract (arXiv)

Lifelong imitation learning for manipulation tasks poses significant challenges due to distribution shifts that occur in incremental learning steps. Existing methods often focus on unsupervised skill discovery to construct an ever-growing skill library or distillation from multiple policies, which can lead to scalability issues as diverse manipulation tasks are continually introduced and may fail to ensure a consistent latent space throughout the learning process, leading to catastrophic forgetting of previously learned skills. In this paper, we introduce M2Distill, a multi-modal distillation-based method for lifelong imitation learning focusing on preserving consistent latent space across vision, language, and action distributions throughout the learning process. By regulating the shifts in latent representations across different modalities from previous to current steps, and reducing discrepancies in Gaussian Mixture Model (GMM) policies between consecutive learning steps, we ensure that the learned policy retains its ability to perform previously learned tasks while seamlessly integrating new skills. Extensive evaluations on the LIBERO lifelong imitation learning benchmark suites, including LIBERO-OBJECT, LIBERO-GOAL, and LIBERO-SPATIAL, demonstrate that our method consistently outperforms prior state-of-the-art methods across all evaluated metrics.