mlsanityData to 7 Oct 2026

Accelerating Dataset Distillation via Model Augmentation

arXiv 2212.06152 ↗First posted 12 Dec 2022cs.LG cs.AIcited by 62

Not on the map. Why?

Abstract (arXiv)

Dataset Distillation (DD), a newly emerging field, aims at generating much smaller but efficient synthetic training datasets from large ones. Existing DD methods based on gradient matching achieve leading performance; however, they are extremely computationally intensive as they require continuously optimizing a dataset among thousands of randomly initialized models. In this paper, we assume that training the synthetic data with diverse models leads to better generalization performance. Thus we propose two model augmentation techniques, i.e. using early-stage models and parameter perturbation to learn an informative synthetic set with significantly reduced training cost. Extensive experiments demonstrate that our method achieves up to 20x speedup and comparable performance on par with state-of-the-art methods.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 58 of its 70 references that are in the map. First posted on arXiv 12 Dec 2022.

Builds on

58 of 70 references.

References · 58

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  8. 2019Contrastive Multiview Codingcited by 1,297
  9. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  11. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  12. 2020Dataset Condensation with Gradient Matchingcited by 510
38 more
  1. 2019Similarity-Preserving Knowledge Distillationcited by 507
  2. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
  3. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  4. 2021Dataset Condensation with Distribution Matchingcited by 362
  5. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  6. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  7. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  8. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  9. 2020Time Series Data Augmentation for Deep Learning: A Surveycited by 225
  10. 2018Dataset Distillationcited by 224
  11. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  12. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  13. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 187
  14. 2021Graph Condensation for Graph Neural Networkscited by 150
  15. 2022Federated Class-Incremental Learningcited by 126
  16. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 119
  17. 2020The Early Phase of Neural Network Trainingcited by 114
  18. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  19. 2019Soft-Label Dataset Distillation and Text Dataset Distillationcited by 106
  20. 2022Dataset Condensation with Contrastive Signalscited by 105
  21. 2019Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Datacited by 103
  22. 2022Condensing Graphs via One-Step Gradient Matchingcited by 100
  23. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  24. 2020Flexible Dataset Distillation: Learn Labels Instead of Imagescited by 91
  25. 2017Critical Learning Periods in Deep Neural Networkscited by 78
  26. 2022DC-BENCH: Dataset Condensation Benchmarkcited by 67
  27. 2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 65
  28. 2022Graph Condensation via Receptive Field Distribution Matchingcited by 60
  29. 2021No One Representation to Rule Them All: Overlapping Features of Training Methodscited by 57
  30. 2020Federated Learning via Synthetic Datacited by 53
  31. 2021Soft-Label Anonymous Gastric X-ray Image Distillationcited by 51
  32. 2020What Can Be Transferred: Unsupervised Domain Adaptation for Endoscopic Lesions Segmentationcited by 50
  33. 2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
  34. 2020On the Generalization Effects of Linear Transformations in Data Augmentationcited by 47
  35. 2022Delving into Effective Gradient Matching for Dataset Condensationcited by 40
  36. 2021Distilled Replay: Overcoming Forgetting through Synthetic Samplescited by 36
  37. 2022Compressed Gastric Image Generation Based on Soft-Label Dataset Distillation for Medical Data Sharingcited by 35
  38. 2022Sample Condensation in Online Continual Learningcited by 31

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 21; 2025: 13; 2026: 7.

62 papers.

Sort:

By area:

  1. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  2. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  3. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  4. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 107
  5. 2023Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matchingcited by 69
  6. 2023M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancycited by 49
  7. 2023Sequential Subset Matching for Dataset Distillationcited by 40
  8. 2024Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustmentcited by 32
  9. 2023AutoReP: Automatic ReLU Replacement for Fast Private Network Inferencecited by 29
  10. 2024Navigating Complexity: Toward Lossless Graph Condensation via Expanding Window Matchingcited by 28
  11. 2023LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inferencecited by 26
  12. 2024Dataset Distillation by Automatic Training Trajectoriescited by 26
38 more
  1. 2025The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutionscited by 24
  2. 2023Accel-GCN: High-Performance GPU Accelerator Design for Graph Convolution Networkscited by 22
  3. 2023You Only Condense Once: Two Rules for Pruning Condensed Datasetscited by 20
  4. 2024A CLIP-Powered Framework for Robust and Generalizable Data Selectioncited by 20
  5. 2024Multisize Dataset Condensationcited by 19
  6. 2024Graph Data Condensation via Self-expressive Graph Structure Reconstructioncited by 18
  7. 2024Low-Rank Similarity Mining for Multimodal Dataset Distillationcited by 16
  8. 2024DANCE: Dual-View Distribution Alignment for Dataset Condensationcited by 15
  9. 2023Rethinking Data Distillation: Do Not Overlook Calibrationcited by 13
  10. 2023Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraintscited by 13
  11. 2024Dataset Quantization with Active Learning based Adaptive Samplingcited by 13
  12. 2024Teddy: Efficient Large-Scale Dataset Distillation via Taylor-Approximated Matchingcited by 13
  13. 2025Does Training with Synthetic Data Truly Protect Privacy?cited by 12
  14. 2023QuickDrop: Efficient Federated Unlearning by Integrated Dataset Distillationcited by 11
  15. 2024Adaptive Dataset Quantizationcited by 9
  16. 2024Dataset Condensation for Time Series Classification via Dual Domain Matchingcited by 8
  17. 2023Dataset Distillation in Latent Spacecited by 7
  18. 2024CondTSF: One-line Plugin of Dataset Condensation for Time Series Forecastingcited by 6
  19. 2025Unifying Dataset Pruning and Distillation for Efficient Large-scale Compressioncited by 6
  20. 2023Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correctioncited by 5
  21. 2023Embarassingly Simple Dataset Distillationcited by 5
  22. 2024One Category One Prompt: Dataset Distillation using Diffusion Modelscited by 5
  23. 2024Label-Augmented Dataset Distillationcited by 5
  24. 2023Bayesian Pseudo-Coresets via Contrastive Divergencecited by 4
  25. 2023FLTracer: Accurate Poisoning Attack Provenance in Federated Learningcited by 4
  26. 2023Filter & Align: Leveraging Human Knowledge to Curate Image-Text Datacited by 4
  27. 2024Exploring the Impact of Dataset Bias on Dataset Distillationcited by 4
  28. 2024Dataset Distillation for Histopathology Image Classificationcited by 4
  29. 2024Decomposed Distribution Matching in Dataset Condensationcited by 3
  30. 2025FedWSIDD: Federated Whole Slide Image Classification via Dataset Distillationcited by 3
  31. 2025RL-Selector: Reinforcement Learning-Guided Data Selection via Redundancy Assessmentcited by 3
  32. 2026Multimodal Distribution Matching for Vision-Language Dataset Distillationcited by 3
  33. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 3
  34. 2024Efficiency for Free: Ideal Data Are Transportable Representationscited by 2
  35. 2025Understanding Dataset Distillation via Spectral Filteringcited by 2
  36. 2024DDFAD: Dataset Distillation Framework for Audio Datacited by 1
  37. 2025DAM: Domain-Aware Module for Multi-Domain Dataset Condensationcited by 1
  38. 2025Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillationcited by 1

50 most cited of 62 All 62 (JSON)