mlsanityData to 14 Oct 2026

Dataset Condensation with Distribution Matching

arXiv 2110.04181 ↗First posted 8 Oct 2021cs.LG cs.CVcited by 365

Not on the map. Why?

Cited by 25 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Computational cost of training state-of-the-art deep models in many learning problems is rapidly increasing due to more sophisticated models and larger datasets. A recent promising direction for reducing training cost is dataset condensation that aims to replace the original large training set with a significantly smaller learned synthetic set while preserving the original information. While training deep models on the small set of condensed images can be extremely fast, their synthesis remains computationally expensive due to the complex bi-level optimization and second-order derivative computation. In this work, we propose a simple yet effective method that synthesizes condensed images by matching feature distributions of the synthetic and original training images in many sampled embedding spaces. Our method significantly reduces the synthesis cost while achieving comparable or better performance. Thanks to its efficiency, we apply our method to more realistic and larger datasets with sophisticated neural architectures and obtain a significant performance boost. We also show promising practical benefits of our method in continual learning and neural architecture search.

Key facts. Cited by 365 papers in the map (data through 14 Oct 2026). Builds on 28 of its 56 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

28 of 56 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2013Auto-Encoding Variational Bayescited by 4,053
  5. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  6. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  7. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  8. 2014Conditional Generative Adversarial Netscited by 2,162
  9. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  10. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  11. 2018End-to-End Incremental Learningcited by 597
  12. 2019Gradient based sample selection for online continual learningcited by 571
16 more
  1. 2020Dataset Condensation with Gradient Matchingcited by 516
  2. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 363
  3. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 344
  4. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 291
  5. 2015Generative Moment Matching Networkscited by 257
  6. 2018Dataset Distillationcited by 230
  7. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  8. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
  9. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 192
  10. 2020Threats to Federated Learning: A Surveycited by 178
  11. 2019Soft-Label Dataset Distillation and Text Dataset Distillationcited by 107
  12. 2019Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Datacited by 104
  13. 2020Flexible Dataset Distillation: Learn Labels Instead of Imagescited by 91
  14. 2020Dual Contradistinctive Generative Autoencodercited by 58
  15. 2020ScaIL: Classifier Weights Scaling for Class Incremental Learningcited by 47
  16. 2015Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?cited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 25; 2023: 72; 2024: 96; 2025: 103; 2026: 69.

365 papers.

Sort:

By area:

  1. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
  2. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 188
  3. 2022Scaling Up Dataset Distillation to ImageNet-1K with Constant Memorycited by 181
  4. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  5. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  6. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 138
  7. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 133
  8. 2023Dataset Distillation: A Comprehensive Reviewcited by 127
  9. 2023Generalizing Dataset Distillation via Deep Generative Priorcited by 126
  10. 2023Improved Distribution Matching for Dataset Condensationcited by 124
  11. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 120
  12. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
38 more
  1. 2022Dataset Distillation via Factorizationcited by 115
  2. 2023A Comprehensive Survey of Dataset Distillationcited by 115
  3. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  4. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 109
  5. 2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
  6. 2022Condensing Graphs via One-Step Gradient Matchingcited by 100
  7. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 94
  8. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  9. 2023Data Distillation: A Surveycited by 87
  10. 2023DataDAM: Efficient Dataset Distillation with Attention Matchingcited by 84
  11. 2023Efficient Dataset Distillation via Minimax Diffusioncited by 83
  12. 2022Synthesizing Informative Training Samples with GANcited by 77
  13. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 77
  14. 2023Structure-free Graph Condensation: From Large-scale Graphs to Condensed Graph-free Datacited by 71
  15. 2023Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matchingcited by 69
  16. 2022DC-BENCH: Dataset Condensation Benchmarkcited by 67
  17. 2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 66
  18. 2023Large-scale Dataset Pruning with Dynamic Uncertaintycited by 63
  19. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  20. 2022Graph Condensation via Receptive Field Distribution Matchingcited by 60
  21. 2024A Comprehensive Survey on Graph Reduction: Sparsification, Coarsening, and Condensationcited by 57
  22. 2023CaT: Balanced Continual Graph Learning with Graph Condensationcited by 52
  23. 2023Dataset Quantizationcited by 48
  24. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  25. 2023DiM: Distilling Dataset into Generative Modelcited by 46
  26. 2023Does Graph Distillation See Like Vision Dataset Counterpart?cited by 45
  27. 2025Dataset Distillation with Neural Characteristic Function: A Minmax Perspectivecited by 45
  28. 2023Sequential Subset Matching for Dataset Distillationcited by 40
  29. 2024Elucidating the Design Space of Dataset Condensationcited by 40
  30. 2023Real-Fake: Effective Training Data Synthesis Through Distribution Matchingcited by 36
  31. 2023Fast Graph Condensation with Structure-based Neural Tangent Kernelcited by 36
  32. 2022Dataset Condensation with Latent Space Knowledge Factorization and Sharingcited by 34
  33. 2022Private Set Generation with Discriminative Informationcited by 34
  34. 2023Loss-Curvature Matching for Dataset Selection and Condensationcited by 33
  35. 2024An Aggregation-Free Federated Learning for Tackling Data Heterogeneitycited by 33
  36. 2024Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustmentcited by 33
  37. 2023A Survey on Dataset Distillation: Approaches, Applications and Future Directionscited by 32
  38. 2023Graph Condensation for Inductive Node Representation Learningcited by 31

50 most cited of 365 All 365 (JSON)