mlsanityData to 7 Oct 2026

Distilled Replay: Overcoming Forgetting through Synthetic Samples

arXiv 2103.15851 ↗First posted 29 Mar 2021cs.LG cs.AIcited by 36

Not on the map. Why?

Abstract (arXiv)

Replay strategies are Continual Learning techniques which mitigate catastrophic forgetting by keeping a buffer of patterns from previous experiences, which are interleaved with new data during training. The amount of patterns stored in the buffer is a critical parameter which largely influences the final performance and the memory footprint of the approach. This work introduces Distilled Replay, a novel replay strategy for Continual Learning which is able to mitigate forgetting by keeping a very small buffer (1 pattern per class) of highly informative samples. Distilled Replay builds the buffer through a distillation process which compresses a large dataset into a tiny set of informative examples. We show the effectiveness of our Distilled Replay against popular replay-based strategies on four Continual Learning benchmarks.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 30 of its 62 references that are in the map. First posted on arXiv 29 Mar 2021.

Builds on

30 of 62 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2016Learning without Forgettingcited by 2,465
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  6. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  7. 2016Progressive Neural Networkscited by 1,332
  8. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  9. 2017Continual Learning with Deep Generative Replaycited by 1,152
  10. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  11. 2018Experience Replay for Continual Learningcited by 1,074
  12. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
18 more
  1. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  2. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  3. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  4. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  5. 2019Three scenarios for continual learningcited by 610
  6. 2020Dataset Condensation with Gradient Matchingcited by 510
  7. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
  8. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  9. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 287
  10. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  11. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  12. 2018Towards Robust Evaluations of Continual Learningcited by 148
  13. 2018Memory Efficient Experience Replay for Streaming Learningcited by 128
  14. 2018Generative replay with feedback connections as a general strategy for continual learningcited by 103
  15. 2019Uncertainty-guided Continual Learning with Bayesian Neural Networkscited by 100
  16. 2019Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challengescited by 92
  17. 2018Progressive Memory Banks for Incremental Domain Adaptationcited by 12
  18. 2020Continual Learning with Gated Incremental Memories for sequential data processingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 14; 2024: 12; 2025: 3; 2026: 1.

36 papers.

Sort:

  1. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  2. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  3. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  4. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  5. 2022Dataset Distillation via Factorizationcited by 115
  6. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  7. 2023Data Distillation: A Surveycited by 86
  8. 2023Structure-free Graph Condensation: From Large-scale Graphs to Condensed Graph-free Datacited by 71
  9. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  10. 2023CaT: Balanced Continual Graph Learning with Graph Condensationcited by 50
  11. 2023Sequential Subset Matching for Dataset Distillationcited by 40
  12. 2024Exploiting Inter-sample and Inter-feature Relations in Dataset Distillationcited by 31
24 more
  1. 2024Navigating Complexity: Toward Lossless Graph Condensation via Expanding Window Matchingcited by 28
  2. 2024Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarioscited by 18
  3. 2024Not All Samples Should Be Utilized Equally: Towards Understanding and Improving Dataset Distillationcited by 14
  4. 2023SeiT: Storage-Efficient Vision Training with Tokens Using 1% of Pixel Storagecited by 8
  5. 2023PUMA: Efficient Continual Graph Learning for Node Classification with Graph Condensationcited by 8
  6. 2024Two Trades is not Baffled: Condensing Graph via Crafting Rational Gradient Matchingcited by 8
  7. 2024DD-RobustBench: An Adversarial Robustness Benchmark for Dataset Distillationcited by 8
  8. 2023The Importance of Robust Features in Mitigating Catastrophic Forgettingcited by 7
  9. 2024Distributional Dataset Distillation with Subtask Decompositioncited by 7
  10. 2023Leveraging Hierarchical Feature Sharing for Efficient Dataset Condensationcited by 6
  11. 2022AdaptCL: Adaptive Continual Learning for Tackling Heterogeneity in Sequential Datasetscited by 4
  12. 2024Prioritize Alignment in Dataset Distillationcited by 4
  13. 2025On Learning Representations for Tabular Data Distillationcited by 4
  14. 2023Boosting the Cross-Architecture Generalization of Dataset Distillation through an Empirical Studycited by 3
  15. 2024Decomposed Distribution Matching in Dataset Condensationcited by 3
  16. 2024Adaptive Hyperparameter Optimization for Continual Learning Scenarioscited by 2
  17. 2024Exemplar-condensed Federated Class-incremental Learningcited by 2
  18. 2022Centroids Matching: an efficient Continual Learning approach operating in the embedding spacecited by 1
  19. 2023Differentially Private Kernel Inducing Points using features from ScatterNets (DP-KIP-ScatterNet) for Privacy Preserving Data Distillationcited by 1
  20. 2023SeiT++: Masked Token Modeling Improves Storage-efficient Trainingcited by 1
  21. 2024One-Shot Collaborative Data Distillationcited by 1
  22. 2026Condensing Large-Scale Datasets Directly with Minimal Information Losscited by 1
  23. 2025Generalized Kernel Inducing Points by Duality Gap for Dataset Distillationcited by 0
  24. 2025Efficient Dataset Distillation through Low-Rank Space Samplingcited by 0