mlsanityData to 7 Oct 2026

Dataset Distillation by Matching Training Trajectories

arXiv 2203.11932 ↗First posted 22 Mar 2022cs.CV cs.AI cs.LGcited by 437

Not on the map. Why?

Cited by 32 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Dataset distillation is the task of synthesizing a small dataset such that a model trained on the synthetic set will match the test accuracy of the model trained on the full dataset. In this paper, we propose a new formulation that optimizes our distilled data to guide networks to a similar state as those trained on real data across many training steps. Given a network, we train it for several iterations on our distilled data and optimize the distilled data with respect to the distance between the synthetically trained parameters and the parameters trained on real data. To efficiently obtain the initial and target network parameters for large-scale datasets, we pre-compute and store training trajectories of expert networks trained on the real dataset. Our method handily outperforms existing methods and also allows us to distill higher-resolution visual data.

Key facts. Cited by 437 papers in the map (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 22 Mar 2022.

Builds on

37 of 49 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  8. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  9. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  10. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  11. 2020Dataset Condensation with Gradient Matchingcited by 510
  12. 2018An Algorithmic Perspective on Imitation Learningcited by 455
25 more
  1. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  2. 2021Dataset Condensation with Distribution Matchingcited by 362
  3. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 361
  4. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  5. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  6. 2019Kornia: an Open Source Differentiable Computer Vision Library for PyTorchcited by 231
  7. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  8. 2018Dataset Distillationcited by 224
  9. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  10. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 191
  11. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 153
  12. 2020Distilled One-Shot Federated Learningcited by 108
  13. 2019Soft-Label Dataset Distillation and Text Dataset Distillationcited by 106
  14. 2019Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Datacited by 103
  15. 2020Flexible Dataset Distillation: Learn Labels Instead of Imagescited by 91
  16. 2020Image Augmentations for GAN Trainingcited by 78
  17. 2017Practical Coreset Constructions for Machine Learningcited by 75
  18. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
  19. 2020Federated Learning via Synthetic Datacited by 53
  20. 2021Soft-Label Anonymous Gastric X-ray Image Distillationcited by 51
  21. 2020Reducing catastrophic forgetting with learning on synthetic datacited by 26
  22. 2020On Data Augmentation for GAN Trainingcited by 25
  23. 2021AMP: Adversarial Motion Priors for Stylized Physics-Based Character Controlcited by 18
  24. 2013Are all training examples equally valuable?cited by 14
  25. 2020SecDD: Efficient and Secure Method for Remotely Training Neural Networkscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 30; 2023: 87; 2024: 123; 2025: 116; 2026: 81.

437 papers.

Sort:

By area:

  1. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 187
  2. 2022Scaling Up Dataset Distillation to ImageNet-1K with Constant Memorycited by 180
  3. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  4. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 171
  5. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  6. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  7. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 136
  8. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
  9. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  10. 2023Generalizing Dataset Distillation via Deep Generative Priorcited by 125
  11. 2023Improved Distribution Matching for Dataset Condensationcited by 124
  12. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
38 more
  1. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 119
  2. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  3. 2022Dataset Distillation via Factorizationcited by 115
  4. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  5. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 107
  6. 2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
  7. 2022Condensing Graphs via One-Step Gradient Matchingcited by 100
  8. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 93
  9. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  10. 2023Data Distillation: A Surveycited by 86
  11. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  12. 2023DataDAM: Efficient Dataset Distillation with Attention Matchingcited by 83
  13. 2023Efficient Dataset Distillation via Minimax Diffusioncited by 81
  14. 2022Synthesizing Informative Training Samples with GANcited by 77
  15. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 77
  16. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 76
  17. 2023Structure-free Graph Condensation: From Large-scale Graphs to Condensed Graph-free Datacited by 71
  18. 2023Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matchingcited by 69
  19. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 68
  20. 2022DC-BENCH: Dataset Condensation Benchmarkcited by 67
  21. 2023Data Pruning via Moving-one-Sample-outcited by 67
  22. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  23. 2023Large-scale Dataset Pruning with Dynamic Uncertaintycited by 63
  24. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  25. 2023Class-Incremental Exemplar Compression for Class-Incremental Learningcited by 50
  26. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  27. 2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
  28. 2023M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancycited by 49
  29. 2023Dataset Quantizationcited by 47
  30. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  31. 2023DiM: Distilling Dataset into Generative Modelcited by 45
  32. 2023Does Graph Distillation See Like Vision Dataset Counterpart?cited by 45
  33. 2025Dataset Distillation with Neural Characteristic Function: A Minmax Perspectivecited by 45
  34. 2023Sequential Subset Matching for Dataset Distillationcited by 40
  35. 2024Elucidating the Design Space of Dataset Condensationcited by 40
  36. 2023An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learningcited by 39
  37. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 38
  38. 2023Knowledge Distillation on Graphs: A Surveycited by 38

50 most cited of 437 All 437 (JSON)