mlsanityData to 7 Oct 2026

Dataset Condensation with Differentiable Siamese Augmentation

arXiv 2102.08259 ↗First posted 16 Feb 2021cs.LG cs.CVcited by 290

Not on the map. Why?

Abstract (arXiv)

In many machine learning problems, large-scale datasets have become the de-facto standard to train state-of-the-art deep networks at the price of heavy computation load. In this paper, we focus on condensing large training sets into significantly smaller synthetic sets which can be used to train deep neural networks from scratch with minimum drop in performance. Inspired from the recent training set synthesis methods, we propose Differentiable Siamese Augmentation that enables effective use of data augmentation to synthesize more informative synthetic images and thus achieves better performance when training networks with augmentations. Experiments on multiple image classification benchmarks demonstrate that the proposed method obtains substantial gains over the state-of-the-art, 7% improvements on CIFAR10 and CIFAR100 datasets. We show with only less than 1% data that our method achieves 99.6%, 94.9%, 88.5%, 71.5% relative performance on MNIST, FashionMNIST, SVHN, CIFAR10 respectively. We also explore the use of our method in continual learning and neural architecture search, and show promising results.

Key facts. Cited by 290 papers in the map (data through 7 Oct 2026). Builds on 44 of its 62 references that are in the map. First posted on arXiv 16 Feb 2021.

Builds on

44 of 62 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  10. 2014Going Deeper with Convolutionscited by 4,231
  11. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
32 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  4. 2014Conditional Generative Adversarial Netscited by 2,152
  5. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  6. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  7. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  8. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  9. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  10. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  11. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  12. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  13. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  14. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  15. 2018End-to-End Incremental Learningcited by 596
  16. 2019Gradient based sample selection for online continual learningcited by 568
  17. 2020Dataset Condensation with Gradient Matchingcited by 510
  18. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  19. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 361
  20. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  21. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  22. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  23. 2018Dataset Distillationcited by 224
  24. 2019Soft-Label Dataset Distillation and Text Dataset Distillationcited by 106
  25. 2019Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Datacited by 103
  26. 2017Learning to Compose Domain-Specific Transformations for Data Augmentationcited by 91
  27. 2020Flexible Dataset Distillation: Learn Labels Instead of Imagescited by 91
  28. 2020Image Augmentations for GAN Trainingcited by 78
  29. 2018Data augmentation instead of explicit regularizationcited by 62
  30. 2020ScaIL: Classifier Weights Scaling for Class Incremental Learningcited by 47
  31. 2019Using Small Proxy Datasets to Accelerate Hyperparameter Searchcited by 13
  32. 2020Augmented Bi-path Network for Few-shot Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 39; 2023: 70; 2024: 83; 2025: 62; 2026: 30.

290 papers.

Sort:

By area:

  1. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
  2. 2021Dataset Condensation with Distribution Matchingcited by 362
  3. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  4. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 187
  5. 2022Scaling Up Dataset Distillation to ImageNet-1K with Constant Memorycited by 180
  6. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  7. 2021Graph Condensation for Graph Neural Networkscited by 150
  8. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 146
  9. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  10. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 136
  11. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
  12. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
38 more
  1. 2023Generalizing Dataset Distillation via Deep Generative Priorcited by 125
  2. 2023Improved Distribution Matching for Dataset Condensationcited by 124
  3. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 119
  4. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  5. 2022Dataset Distillation via Factorizationcited by 115
  6. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  7. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  8. 2022Dataset Condensation with Contrastive Signalscited by 105
  9. 2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
  10. 2022Condensing Graphs via One-Step Gradient Matchingcited by 100
  11. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 93
  12. 2023Data Distillation: A Surveycited by 86
  13. 2023DataDAM: Efficient Dataset Distillation with Attention Matchingcited by 83
  14. 2023Efficient Dataset Distillation via Minimax Diffusioncited by 81
  15. 2022Synthesizing Informative Training Samples with GANcited by 77
  16. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 76
  17. 2023Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matchingcited by 69
  18. 2022DC-BENCH: Dataset Condensation Benchmarkcited by 67
  19. 2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 65
  20. 2023Large-scale Dataset Pruning with Dynamic Uncertaintycited by 63
  21. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  22. 2022Expanding Small-Scale Datasets with Guided Imaginationcited by 61
  23. 2022Graph Condensation via Receptive Field Distribution Matchingcited by 60
  24. 2023CaT: Balanced Continual Graph Learning with Graph Condensationcited by 50
  25. 2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
  26. 2023M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancycited by 49
  27. 2023Dataset Quantizationcited by 47
  28. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  29. 2023DiM: Distilling Dataset into Generative Modelcited by 45
  30. 2025Dataset Distillation with Neural Characteristic Function: A Minmax Perspectivecited by 45
  31. 2022Delving into Effective Gradient Matching for Dataset Condensationcited by 40
  32. 2023Sequential Subset Matching for Dataset Distillationcited by 40
  33. 2024Elucidating the Design Space of Dataset Condensationcited by 40
  34. 2022Submodularity In Machine Learning and Artificial Intelligencecited by 37
  35. 2021Distilled Replay: Overcoming Forgetting through Synthetic Samplescited by 36
  36. 2023Fast Graph Condensation with Structure-based Neural Tangent Kernelcited by 36
  37. 2022Compressed Gastric Image Generation Based on Soft-Label Dataset Distillation for Medical Data Sharingcited by 35
  38. 2022Dataset Condensation with Latent Space Knowledge Factorization and Sharingcited by 34

50 most cited of 290 All 290 (JSON)