mlsanityData to 7 Oct 2026

Dataset Distillation with Infinitely Wide Convolutional Networks

arXiv 2107.13034 ↗First posted 27 Jul 2021cs.LGcited by 215

Not on the map. Why?

Abstract (arXiv)

The effectiveness of machine learning algorithms arises from being able to extract useful features from large amounts of data. As model and dataset sizes increase, dataset distillation methods that compress large datasets into significantly smaller yet highly performant ones will become valuable in terms of training efficiency and useful feature extraction. To that end, we apply a novel distributed kernel based meta-learning framework to achieve state-of-the-art results for dataset distillation using infinitely wide convolutional neural networks. For instance, using only 10 datapoints (0.02% of original dataset), we obtain over 65% test accuracy on CIFAR-10 image classification task, a dramatic improvement over the previous best test accuracy of 40%. Our state-of-the-art results extend across many other settings for MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and SVHN. Furthermore, we perform some preliminary analyses of our distilled datasets to shed light on how they differ from naturally occurring data.

Key facts. Cited by 215 papers in the map (data through 7 Oct 2026). Builds on 47 of its 64 references that are in the map. First posted on arXiv 27 Jul 2021.

Builds on

47 of 64 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  8. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  9. 2017Deep Neural Networks as Gaussian Processescited by 617
  10. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  11. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  12. 2020Dataset Condensation with Gradient Matchingcited by 510
35 more
  1. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  2. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  3. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  4. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
  5. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  6. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  7. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  8. 2016What makes ImageNet good for transfer learning?cited by 192
  9. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 191
  10. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  11. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  12. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  13. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  14. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  15. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  16. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  17. 2020What shapes feature representations? Exploring datasets, architectures, and trainingcited by 125
  18. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  19. 2019Soft-Label Dataset Distillation and Text Dataset Distillationcited by 106
  20. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  21. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  22. 2020Flexible Dataset Distillation: Learn Labels Instead of Imagescited by 91
  23. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  24. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  25. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  26. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  27. 2020Neural Kernels Without Tangentscited by 56
  28. 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
  29. 2020On the training dynamics of deep networks with L2 regularizationcited by 38
  30. 2021Scaling Neural Tangent Kernels via Sketching and Random Featurescited by 32
  31. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
  32. 2020Towards NNGP-guided Neural Architecture Searchcited by 24
  33. 2019Disentangling Trainability and Generalization in Deep Neural Networkscited by 16
  34. 2020Asymptotics of Wide Convolutional Neural Networkscited by 15
  35. 2020Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limitcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 39; 2023: 56; 2024: 59; 2025: 41; 2026: 18.

215 papers.

Sort:

By area:

  1. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
  2. 2021Dataset Condensation with Distribution Matchingcited by 362
  3. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  4. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 187
  5. 2022Scaling Up Dataset Distillation to ImageNet-1K with Constant Memorycited by 180
  6. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  7. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  8. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 136
  9. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
  10. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  11. 2023Generalizing Dataset Distillation via Deep Generative Priorcited by 125
  12. 2023Improved Distribution Matching for Dataset Condensationcited by 124
38 more
  1. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 119
  2. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  3. 2022Dataset Distillation via Factorizationcited by 115
  4. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  5. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  6. 2022Dataset Condensation with Contrastive Signalscited by 105
  7. 2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
  8. 2022Condensing Graphs via One-Step Gradient Matchingcited by 100
  9. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 93
  10. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  11. 2023Data Distillation: A Surveycited by 86
  12. 2023DataDAM: Efficient Dataset Distillation with Attention Matchingcited by 83
  13. 2023Efficient Dataset Distillation via Minimax Diffusioncited by 81
  14. 2022Synthesizing Informative Training Samples with GANcited by 77
  15. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 76
  16. 2023Structure-free Graph Condensation: From Large-scale Graphs to Condensed Graph-free Datacited by 71
  17. 2022DC-BENCH: Dataset Condensation Benchmarkcited by 67
  18. 2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 65
  19. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  20. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  21. 2024A Comprehensive Survey on Graph Reduction: Sparsification, Coarsening, and Condensationcited by 57
  22. 2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
  23. 2023M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancycited by 49
  24. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  25. 2023DiM: Distilling Dataset into Generative Modelcited by 45
  26. 2023Does Graph Distillation See Like Vision Dataset Counterpart?cited by 45
  27. 2022Information-theoretic Online Memory Selection for Continual Learningcited by 42
  28. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  29. 2023Sequential Subset Matching for Dataset Distillationcited by 40
  30. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  31. 2022Submodularity In Machine Learning and Artificial Intelligencecited by 37
  32. 2022Dataset Condensation with Latent Space Knowledge Factorization and Sharingcited by 34
  33. 2023A Survey on Dataset Distillation: Approaches, Applications and Future Directionscited by 32
  34. 2024Exploiting Inter-sample and Inter-feature Relations in Dataset Distillationcited by 31
  35. 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
  36. 2023Frequency Domain-based Dataset Distillationcited by 28
  37. 2024Navigating Complexity: Toward Lossless Graph Condensation via Expanding Window Matchingcited by 28
  38. 2022Dataset Distillation Using Parameter Pruningcited by 27

50 most cited of 215 All 215 (JSON)