mlsanityData to 14 Oct 2026

Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory

arXiv 2211.10586 ↗First posted 19 Nov 2022cs.CV cs.AIcited by 181

Not on the map. Why?

Abstract (arXiv)

Dataset Distillation is a newly emerging area that aims to distill large datasets into much smaller and highly informative synthetic ones to accelerate training and reduce storage. Among various dataset distillation methods, trajectory-matching-based methods (MTT) have achieved SOTA performance in many tasks, e.g., on CIFAR-10/100. However, due to exorbitant memory consumption when unrolling optimization through SGD steps, MTT fails to scale to large-scale datasets such as ImageNet-1K. Can we scale this SOTA method to ImageNet-1K and does its effectiveness on CIFAR transfer to ImageNet-1K? To answer these questions, we first propose a procedure to exactly compute the unrolled gradient with constant memory complexity, which allows us to scale MTT to ImageNet-1K seamlessly with ~6x reduction in memory footprint. We further discover that it is challenging for MTT to handle datasets with a large number of classes, and propose a novel soft label assignment that drastically improves its convergence. The resulting algorithm sets new SOTA on ImageNet-1K: we can scale up to 50 IPCs (Image Per Class) on ImageNet-1K on a single GPU (all previous methods can only scale to 2 IPCs on ImageNet-1K), leading to the best accuracy (only 5.9% accuracy drop against full dataset training) while utilizing only 4.2% of the number of data points - an 18.2% absolute gain over prior SOTA. Our code is available at https://github.com/justincui03/tesla

Key facts. Cited by 181 papers in the map (data through 14 Oct 2026). Builds on 28 of its 35 references that are in the map. First posted on arXiv 19 Nov 2022.

Builds on

28 of 35 references.

References · 28

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  7. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  8. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  9. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  10. 2018End-to-End Incremental Learningcited by 597
  11. 2020Dataset Condensation with Gradient Matchingcited by 516
  12. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
16 more
  1. 2021Dataset Condensation with Distribution Matchingcited by 365
  2. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 363
  3. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 291
  4. 2019Kornia: an Open Source Differentiable Computer Vision Library for PyTorchcited by 232
  5. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  6. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
  7. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 188
  8. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  9. 2020Distilled One-Shot Federated Learningcited by 109
  10. 2022Dataset Condensation with Contrastive Signalscited by 105
  11. 2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
  12. 2021Rethinking Architecture Selection in Differentiable NAScited by 92
  13. 2020Flexible Dataset Distillation: Learn Labels Instead of Imagescited by 91
  14. 2022DC-BENCH: Dataset Condensation Benchmarkcited by 67
  15. 2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 66
  16. 2020On Data Augmentation for GAN Trainingcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 31; 2024: 59; 2025: 55; 2026: 36.

181 papers.

Sort:

By area:

  1. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 138
  2. 2023Dataset Distillation: A Comprehensive Reviewcited by 127
  3. 2023Generalizing Dataset Distillation via Deep Generative Priorcited by 126
  4. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  5. 2023A Comprehensive Survey of Dataset Distillationcited by 115
  6. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 109
  7. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 94
  8. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  9. 2023Data Distillation: A Surveycited by 87
  10. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 77
  11. 2023Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matchingcited by 69
  12. 2023Large-scale Dataset Pruning with Dynamic Uncertaintycited by 63
38 more
  1. 2025Dataset Distillation with Neural Characteristic Function: A Minmax Perspectivecited by 45
  2. 2023Sequential Subset Matching for Dataset Distillationcited by 40
  3. 2024Elucidating the Design Space of Dataset Condensationcited by 40
  4. 2024Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustmentcited by 33
  5. 2023Dataset Distillation via the Wasserstein Metriccited by 31
  6. 2024A Label is Worth a Thousand Images in Dataset Distillationcited by 31
  7. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 27
  8. 2023Too Large; Data Reduction for Vision-Language Pre-Trainingcited by 26
  9. 2023Dataset Distillation via Curriculum Data Synthesis in Large Data Eracited by 26
  10. 2024Dataset Distillation by Automatic Training Trajectoriescited by 26
  11. 2025The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutionscited by 26
  12. 2024Multi-modal Vision Pre-training for Medical Image Analysiscited by 23
  13. 2024SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matchingcited by 22
  14. 2023Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglementcited by 21
  15. 2023MIM4DD: Mutual Information Maximization for Dataset Distillationcited by 21
  16. 2024Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchorscited by 21
  17. 2024Towards Stable and Storage-efficient Dataset Distillation: Matching Convexified Trajectorycited by 21
  18. 2023You Only Condense Once: Two Rules for Pruning Condensed Datasetscited by 20
  19. 2024DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillationcited by 20
  20. 2024DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillationcited by 20
  21. 2023Vision-Language Dataset Distillationcited by 19
  22. 2024Multisize Dataset Condensationcited by 19
  23. 2024Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation?cited by 19
  24. 2024Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarioscited by 18
  25. 2025Taming Diffusion for Dataset Distillation with High Representativenesscited by 18
  26. 2024Towards Adversarially Robust Dataset Distillation by Curvature Regularizationcited by 17
  27. 2024Low-Rank Similarity Mining for Multimodal Dataset Distillationcited by 17
  28. 2023Dataset Condensation via Generative Modelcited by 15
  29. 2025Hyperbolic Dataset Distillationcited by 15
  30. 2023Towards Trustworthy Dataset Distillationcited by 14
  31. 2024Not All Samples Should Be Utilized Equally: Towards Understanding and Improving Dataset Distillationcited by 14
  32. 2025Dataset Distillation via Committee Votingcited by 14
  33. 2025CaO2: Rectifying Inconsistencies in Diffusion-Based Dataset Distillationcited by 14
  34. 2024What is Dataset Distillation Learning?cited by 13
  35. 2024Hierarchical Features Matter: A Deep Exploration of Progressive Parameterization Method for Dataset Distillationcited by 13
  36. 2024Dataset Quantization with Active Learning based Adaptive Samplingcited by 13
  37. 2024Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensatorcited by 13
  38. 2024Generative Dataset Distillation Based on Diffusion Modelcited by 13

50 most cited of 181 All 181 (JSON)