# Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning

arXiv: https://arxiv.org/abs/2603.03818 · HTML: https://mlsanity.com/paper/2603.03818/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models.
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Placed in

- Vision-language-action models / [Reinforcement learning and post-training](https://mlsanity.com/vla/reinforcement-learning-and-post-training/)

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [Towards Continual Knowledge Learning of Language Models](https://mlsanity.com/paper/2110.03215/) (2021)
- [Fine-tuned Language Models are Continual Learners](https://mlsanity.com/paper/2205.12393/) (2022)
- [TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models](https://mlsanity.com/paper/2204.14211/) (2022)
- [RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation](https://mlsanity.com/paper/2306.11706/) (2023)
- [LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discovery](https://mlsanity.com/paper/2311.02058/) (2023)
- [Drinking from a Firehose: Continual Learning with Web-scale Natural Language](https://mlsanity.com/paper/2007.09335/) (2020)
- [Robust Finetuning of Vision-Language-Action Robot Policies via Parameter Merging](https://mlsanity.com/paper/2512.08333/) (2025)

## Built on by (most cited first, 17 of 17)

- [Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning](https://mlsanity.com/paper/2603.11653/) (2026)
- [Forget to Improve: On-Device LLM-Agent Continual Learning via Budget-Curated Memory](https://mlsanity.com/paper/2606.25115/) (2026)
- [Can Vision-Language-Action Models Learn from Real-World Data Continually without Forgetting?](https://mlsanity.com/paper/2605.26820/) (2026)
- [World Action Models Enable Continual Imitation Learning with Recurrent Generative Replays](https://mlsanity.com/paper/2606.27374/) (2026)
- [PHASER: Phase-Aware and Semantic Experience Replay for Vision-Language-Action Models](https://mlsanity.com/paper/2606.03598/) (2026)
- [Towards Human-like Physical Intelligence: Lifelong Vision-Language-Action Learning for Robotic Manipulation](https://mlsanity.com/paper/2607.14852/) (2026)
- [Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFT](https://mlsanity.com/paper/2605.08879/) (2026)
- [Suppression Sticks, Locality Is Fragile: A Closed-Loop Target-and-Control Audit of Task-Vector Negation in VLA Policies](https://mlsanity.com/paper/2608.04692/) (2026)
- [Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priors](https://mlsanity.com/paper/2606.25527/) (2026)
- [Fine-Tuning VLAs with Self-Demonstrated Generative Control for Multi-Task Manipulation](https://mlsanity.com/paper/2608.19490/) (2026)
- [OrthoSkillVLA: Continual Skill Learning via Gradient-Informed Skill Subspace Adaptation](https://mlsanity.com/paper/2608.19589/) (2026)
- [FAN: Foresight Action Normalization for Continual Adaptation of Vision-Language-Action Models](https://mlsanity.com/paper/2609.21358/) (2026)
- [SAMBAR: Selective Anchoring via Method of Multipliers for Balanced Knowledge Acquisition and Retention in Vision-Language-Action Models](https://mlsanity.com/paper/2609.32108/) (2026)
- [Train Together or Merge Later? Unifying VLA Experts via a Shared Action Interface](https://mlsanity.com/paper/2609.33125/) (2026)
- [Inline Memory Meets Reusable Skills: Memory-centric Framework for Vision-Language-Action Model](https://mlsanity.com/paper/2609.39794/) (2026)
- [Data-Free Continual Learning for World Action Model with Rectified Generative Replay](https://mlsanity.com/paper/2610.18038/) (2026)
- [ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models](https://mlsanity.com/paper/2610.18596/) (2026)

## Abstract (arXiv)

Continual learning is a long-standing challenge in robot policy learning, where a policy must acquire new skills over time without catastrophically forgetting previously learned ones. While prior work has extensively studied continual learning in relatively small behavior cloning (BC) policy models trained from scratch, its behavior in modern large-scale pretrained Vision-Language-Action (VLA) models remains underexplored. In this work, we found that pretrained VLAs are remarkably resistant to forgetting compared with smaller policy models trained from scratch. Simple Experience Replay (ER) works surprisingly well on VLAs, sometimes achieving zero forgetting even with a small replay data size. Our analysis reveals that pretraining plays a critical role in downstream continual learning performance: large pretrained models mitigate forgetting with a small replay buffer size while maintaining strong forward learning capabilities. Furthermore, we found that VLAs can retain relevant knowledge from prior tasks despite performance degradation during learning new tasks. This knowledge retention enables rapid recovery of seemingly forgotten skills through finetuning. Together, these insights imply that large-scale pretraining fundamentally changes the dynamics of continual learning, enabling models to continually acquire new skills over time with simple replay. Code and more information can be found at https://continual-vlas.github.io/forget-me-not/