# Progress & Compress: A scalable framework for continual learning

arXiv: https://arxiv.org/abs/1805.06370 · HTML: https://mlsanity.com/paper/1805.06370/

## Key facts

- Cited by 520 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2018.

## Built on by (most cited first, 30 of 520)

- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Mitigating the Alignment Tax of RLHF](https://mlsanity.com/paper/2309.06256/) (2023)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Continual Learning and Private Unlearning](https://mlsanity.com/paper/2203.12817/) (2022)
- [REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2505.24760/) (2025)
- [Understanding and Preventing Capacity Loss in Reinforcement Learning](https://mlsanity.com/paper/2204.09560/) (2022)
- [Avalanche: an End-to-End Library for Continual Learning](https://mlsanity.com/paper/2104.00405/) (2021)
- [Continual World: A Robotic Benchmark For Continual Reinforcement Learning](https://mlsanity.com/paper/2105.10919/) (2021)
- [Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora](https://mlsanity.com/paper/2110.08534/) (2021)
- [SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding](https://mlsanity.com/paper/2310.15308/) (2023)
- [A Definition of Continual Reinforcement Learning](https://mlsanity.com/paper/2307.11046/) (2023)
- [Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations](https://mlsanity.com/paper/2103.06342/) (2021)
- [Representational Continuity for Unsupervised Continual Learning](https://mlsanity.com/paper/2110.06976/) (2021)
- [Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time](https://mlsanity.com/paper/2211.14238/) (2022)
- [Continual Learning of Natural Language Processing Tasks: A Survey](https://mlsanity.com/paper/2211.12701/) (2022)
- [TRGP: Trust Region Gradient Projection for Continual Learning](https://mlsanity.com/paper/2202.02931/) (2022)
- [Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks](https://mlsanity.com/paper/2112.10017/) (2021)
- [Continual Backprop: Stochastic Gradient Descent with Persistent Randomness](https://mlsanity.com/paper/2108.06325/) (2021)
- [A Theoretical Study on Solving Continual Learning](https://mlsanity.com/paper/2211.02633/) (2022)
- [Model Merging by Uncertainty-Based Gradient Matching](https://mlsanity.com/paper/2310.12808/) (2023)
- [Continual Learning and Catastrophic Forgetting](https://mlsanity.com/paper/2403.05175/) (2024)
- [AFEC: Active Forgetting of Negative Transfer in Continual Learning](https://mlsanity.com/paper/2110.12187/) (2021)
- [ELLE: Efficient Lifelong Pre-training for Emerging Data](https://mlsanity.com/paper/2203.06311/) (2022)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [Continual Learning via Local Module Composition](https://mlsanity.com/paper/2111.07736/) (2021)

## Abstract (arXiv)

We introduce a conceptually simple and scalable framework for continual learning domains where tasks are learned sequentially. Our method is constant in the number of parameters and is designed to preserve performance on previously encountered tasks while accelerating learning progress on subsequent problems. This is achieved by training a network with two components: A knowledge base, capable of solving previously encountered problems, which is connected to an active column that is employed to efficiently learn the current task. After learning a new task, the active column is distilled into the knowledge base, taking care to protect any previously acquired skills. This cycle of active learning (progression) followed by consolidation (compression) requires no architecture growth, no access to or storing of previous data or tasks, and no task-specific parameters. We demonstrate the progress & compress approach on sequential classification of handwritten alphabets as well as two reinforcement learning domains: Atari games and 3D maze navigation.