# Condensed Composite Memory Continual Learning

arXiv: https://arxiv.org/abs/2102.09890 · HTML: https://mlsanity.com/paper/2102.09890/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 19 Feb 2021.

## Builds on (most cited first, 17 of 17)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation](https://mlsanity.com/paper/1811.10959/) (2018)
- [Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines](https://mlsanity.com/paper/1810.12488/) (2018)
- [REMIND Your Neural Network to Prevent Catastrophic Forgetting](https://mlsanity.com/paper/1910.02509/) (2019)
- [Coresets via Bilevel Optimization for Continual Learning and Streaming](https://mlsanity.com/paper/2006.03875/) (2020)
- [Generative replay with feedback connections as a general strategy for continual learning](https://mlsanity.com/paper/1809.10635/) (2018)
- [Latent Replay for Real-Time Continual Learning](https://mlsanity.com/paper/1912.01100/) (2019)
- [Learning to Continually Learn](https://mlsanity.com/paper/2002.09571/) (2020)

## Built on by (most cited first, 20 of 20)

- [Dataset Distillation: A Comprehensive Review](https://mlsanity.com/paper/2301.07014/) (2023)
- [Dataset Distillation via Factorization](https://mlsanity.com/paper/2210.16774/) (2022)
- [A Comprehensive Survey of Dataset Distillation](https://mlsanity.com/paper/2301.05603/) (2023)
- [DREAM: Efficient Dataset Distillation by Representative Matching](https://mlsanity.com/paper/2302.14416/) (2023)
- [Data Distillation: A Survey](https://mlsanity.com/paper/2301.04272/) (2023)
- [Synthesizing Informative Training Samples with GAN](https://mlsanity.com/paper/2204.07513/) (2022)
- [Structure-free Graph Condensation: From Large-scale Graphs to Condensed Graph-free Data](https://mlsanity.com/paper/2306.02664/) (2023)
- [CaT: Balanced Continual Graph Learning with Graph Condensation](https://mlsanity.com/paper/2309.09455/) (2023)
- [M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy](https://mlsanity.com/paper/2312.15927/) (2023)
- [Delving into Effective Gradient Matching for Dataset Condensation](https://mlsanity.com/paper/2208.00311/) (2022)
- [Sample Condensation in Online Continual Learning](https://mlsanity.com/paper/2206.11849/) (2022)
- [Dataset Distillation Using Parameter Pruning](https://mlsanity.com/paper/2209.14609/) (2022)
- [Dataset Distillation by Automatic Training Trajectories](https://mlsanity.com/paper/2407.14245/) (2024)
- [Dataset Distillation for Medical Dataset Sharing](https://mlsanity.com/paper/2209.14603/) (2022)
- [DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation](https://mlsanity.com/paper/2404.00264/) (2024)
- [Importance-Aware Adaptive Dataset Distillation](https://mlsanity.com/paper/2401.15863/) (2024)
- [PUMA: Efficient Continual Graph Learning for Node Classification with Graph Condensation](https://mlsanity.com/paper/2312.14439/) (2023)
- [Improve Cross-Architecture Generalization on Dataset Distillation](https://mlsanity.com/paper/2402.13007/) (2024)
- [Boosting the Cross-Architecture Generalization of Dataset Distillation through an Empirical Study](https://mlsanity.com/paper/2312.05598/) (2023)
- [Differentially Private Kernel Inducing Points using features from ScatterNets (DP-KIP-ScatterNet) for Privacy Preserving Data Distillation](https://mlsanity.com/paper/2301.13389/) (2023)

## Abstract (arXiv)

Deep Neural Networks (DNNs) suffer from a rapid decrease in performance when trained on a sequence of tasks where only data of the most recent task is available. This phenomenon, known as catastrophic forgetting, prevents DNNs from accumulating knowledge over time. Overcoming catastrophic forgetting and enabling continual learning is of great interest since it would enable the application of DNNs in settings where unrestricted access to all the training data at any time is not always possible, e.g. due to storage limitations or legal issues. While many recently proposed methods for continual learning use some training examples for rehearsal, their performance strongly depends on the number of stored examples. In order to improve performance of rehearsal for continual learning, especially for a small number of stored examples, we propose a novel way of learning a small set of synthetic examples which capture the essence of a complete dataset. Instead of directly learning these synthetic examples, we learn a weighted combination of shared components for each example that enables a significant increase in memory efficiency. We demonstrate the performance of our method on commonly used datasets and compare it to recently proposed related methods and baselines.