# Reducing catastrophic forgetting with learning on synthetic data

arXiv: https://arxiv.org/abs/2004.14046 · HTML: https://mlsanity.com/paper/2004.14046/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 26 of 26)

- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dataset Distillation: A Comprehensive Review](https://mlsanity.com/paper/2301.07014/) (2023)
- [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023)
- [Improved Distribution Matching for Dataset Condensation](https://mlsanity.com/paper/2307.09742/) (2023)
- [Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching](https://mlsanity.com/paper/2310.05773/) (2023)
- [Dataset Distillation via Factorization](https://mlsanity.com/paper/2210.16774/) (2022)
- [Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching](https://mlsanity.com/paper/2311.17950/) (2023)
- [Accelerating Dataset Distillation via Model Augmentation](https://mlsanity.com/paper/2212.06152/) (2022)
- [CaT: Balanced Continual Graph Learning with Graph Condensation](https://mlsanity.com/paper/2309.09455/) (2023)
- [M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy](https://mlsanity.com/paper/2312.15927/) (2023)
- [Elucidating the Design Space of Dataset Condensation](https://mlsanity.com/paper/2404.13733/) (2024)
- [Navigating Complexity: Toward Lossless Graph Condensation via Expanding Window Matching](https://mlsanity.com/paper/2402.05011/) (2024)
- [Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios](https://mlsanity.com/paper/2410.17193/) (2024)
- [Not All Samples Should Be Utilized Equally: Towards Understanding and Improving Dataset Distillation](https://mlsanity.com/paper/2408.12483/) (2024)
- [CaO2: Rectifying Inconsistencies in Diffusion-Based Dataset Distillation](https://mlsanity.com/paper/2506.22637/) (2025)
- [TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendation](https://mlsanity.com/paper/2502.02854/) (2025)
- [PUMA: Efficient Continual Graph Learning for Node Classification with Graph Condensation](https://mlsanity.com/paper/2312.14439/) (2023)
- [Distilling Long-tailed Datasets](https://mlsanity.com/paper/2408.14506/) (2024)
- [Task-Specific Generative Dataset Distillation with Difficulty-Guided Sampling](https://mlsanity.com/paper/2507.03331/) (2025)
- [Improve Cross-Architecture Generalization on Dataset Distillation](https://mlsanity.com/paper/2402.13007/) (2024)
- [Prioritize Alignment in Dataset Distillation](https://mlsanity.com/paper/2408.03360/) (2024)
- [Exemplar-condensed Federated Class-incremental Learning](https://mlsanity.com/paper/2412.18926/) (2024)
- [Differentially Private Kernel Inducing Points using features from ScatterNets (DP-KIP-ScatterNet) for Privacy Preserving Data Distillation](https://mlsanity.com/paper/2301.13389/) (2023)
- [Towards Efficient Deep Hashing Retrieval: Condensing Your Data via Feature-Embedding Matching](https://mlsanity.com/paper/2305.18076/) (2023)
- [Generalized Kernel Inducing Points by Duality Gap for Dataset Distillation](https://mlsanity.com/paper/2502.12607/) (2025)
- [GLOBE: Trajectory-Aligned Gradient Matching with Structured SparseOptimization for Coreset Selection](https://mlsanity.com/paper/2608.02690/) (2026)

## Abstract (arXiv)

Catastrophic forgetting is a problem caused by neural networks' inability to learn data in sequence. After learning two tasks in sequence, performance on the first one drops significantly. This is a serious disadvantage that prevents many deep learning applications to real-life problems where not all object classes are known beforehand; or change in data requires adjustments to the model. To reduce this problem we investigate the use of synthetic data, namely we answer a question: Is it possible to generate such data synthetically which learned in sequence does not result in catastrophic forgetting? We propose a method to generate such data in two-step optimisation process via meta-gradients. Our experimental results on Split-MNIST dataset show that training a model on such synthetic data in sequence does not result in catastrophic forgetting. We also show that our method of generating data is robust to different learning scenarios.