mlsanityData to 7 Oct 2026

Efficient Dataset Distillation Using Random Feature Approximation

arXiv 2210.12067 ↗First posted 21 Oct 2022cs.LG cs.AI cs.NEcited by 111

Not on the map. Why?

Abstract (arXiv)

Dataset distillation compresses large datasets into smaller synthetic coresets which retain performance with the aim of reducing the storage and computational burden of processing the entire dataset. Today's best-performing algorithm, textitKernel Inducing Points (KIP), which makes use of the correspondence between infinite-width neural networks and kernel-ridge regression, is prohibitively slow due to the exact computation of the neural tangent kernel matrix, scaling O(|S|2), with |S| being the coreset size. To improve this, we propose a novel algorithm that uses a random feature approximation (RFA) of the Neural Network Gaussian Process (NNGP) kernel, which reduces the kernel matrix computation to O(|S|). Our algorithm provides at least a 100-fold speedup over KIP and can run on a single GPU. Our new method, termed an RFA Distillation (RFAD), performs competitively with KIP and other dataset condensation algorithms in accuracy over a range of large-scale datasets, both in kernel regression and finite-width network training. We demonstrate the effectiveness of our approach on tasks involving model interpretability and privacy preservation.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Builds on 49 of its 78 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

49 of 78 references.

References · 49

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  3. 2016Deep Learning with Differential Privacycited by 2,682
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  6. 2016Matching Networks for One Shot Learningcited by 2,246
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  8. 2018On Lazy Training in Differentiable Programmingcited by 671
  9. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  10. 2017Deep Neural Networks as Gaussian Processescited by 617
  11. 2018End-to-End Incremental Learningcited by 596
  12. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
37 more
  1. 2019Gradient based sample selection for online continual learningcited by 568
  2. 2020Dataset Condensation with Gradient Matchingcited by 510
  3. 2021Dataset Condensation with Distribution Matchingcited by 362
  4. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 361
  5. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  6. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  7. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  8. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  9. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  10. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  11. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  12. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 191
  13. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  14. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  15. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  16. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  17. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  18. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  19. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  20. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  21. 2019Soft-Label Dataset Distillation and Text Dataset Distillationcited by 106
  22. 2020Flexible Dataset Distillation: Learn Labels Instead of Imagescited by 91
  23. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  24. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  25. 2016Coresets and Sketchescited by 63
  26. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  27. 2020Neural Kernels Without Tangentscited by 56
  28. 2018Dirichlet-based Gaussian Processes for Large-scale Calibrated Classificationcited by 50
  29. 2020ScaIL: Classifier Weights Scaling for Class Incremental Learningcited by 47
  30. 2015Input Sparsity Time Low-Rank Approximation via Ridge Leverage Score Samplingcited by 46
  31. 2017Training Gaussian Mixture Models at Scale via Coresetscited by 38
  32. 2021Scaling Neural Tangent Kernels via Sketching and Random Featurescited by 32
  33. 2020Towards NNGP-guided Neural Architecture Searchcited by 24
  34. 2019Introduction to Coresets: Accurate Coresetscited by 17
  35. 2019Using Small Proxy Datasets to Accelerate Hyperparameter Searchcited by 13
  36. 2018Response Characterization for Auditing Cell Dynamics in Long Short-term Memory Networkscited by 10
  37. 2022Interpreting Neural Policies with Disentangled Tree Representationscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 36; 2024: 30; 2025: 29; 2026: 16.

111 papers.

Sort:

By area:

  1. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 136
  2. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  3. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  4. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  5. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 107
  6. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 93
  7. 2023Data Distillation: A Surveycited by 86
  8. 2023Efficient Dataset Distillation via Minimax Diffusioncited by 81
  9. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 76
  10. 2023M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancycited by 49
  11. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  12. 2023Sequential Subset Matching for Dataset Distillationcited by 40
38 more
  1. 2023A Survey on Dataset Distillation: Approaches, Applications and Future Directionscited by 32
  2. 2024Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustmentcited by 32
  3. 2023Dataset Distillation via the Wasserstein Metriccited by 31
  4. 2023Dataset Distillation via Curriculum Data Synthesis in Large Data Eracited by 25
  5. 2024Multi-modal Vision Pre-training for Medical Image Analysiscited by 22
  6. 2023Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglementcited by 21
  7. 2023Leave-one-out Distinguishability in Machine Learningcited by 20
  8. 2023You Only Condense Once: Two Rules for Pruning Condensed Datasetscited by 20
  9. 2024Towards Stable and Storage-efficient Dataset Distillation: Matching Convexified Trajectorycited by 20
  10. 2023Summarizing Stream Data for Memory-Constrained Online Continual Learningcited by 19
  11. 2024Multisize Dataset Condensationcited by 19
  12. 2024Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation?cited by 19
  13. 2024Importance-Aware Adaptive Dataset Distillationcited by 18
  14. 2024Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarioscited by 18
  15. 2023Data Distillation Can Be Like Vodka: Distilling More Times For Better Qualitycited by 17
  16. 2025Dataset Distillation via Vision-Language Category Prototypecited by 17
  17. 2024Low-Rank Similarity Mining for Multimodal Dataset Distillationcited by 16
  18. 2024DANCE: Dual-View Distribution Alignment for Dataset Condensationcited by 15
  19. 2023Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillationcited by 14
  20. 2023Towards Trustworthy Dataset Distillationcited by 14
  21. 2023Rethinking Data Distillation: Do Not Overlook Calibrationcited by 13
  22. 2023Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraintscited by 13
  23. 2024Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensatorcited by 13
  24. 2024Generative Dataset Distillation Based on Diffusion Modelcited by 13
  25. 2025Dataset Distillation via Committee Votingcited by 13
  26. 2025FADRM: Fast and Accurate Data Residual Matching for Dataset Distillationcited by 11
  27. 2023Rethinking Backdoor Attacks on Dataset Distillation: A Kernel Method Perspectivecited by 10
  28. 2024CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognitioncited by 10
  29. 2023A Comprehensive Study on Dataset Distillation: Performance, Privacy, Robustness and Fairnesscited by 9
  30. 2024Graph Condensation for Open-World Graph Learningcited by 9
  31. 2024Distilling Long-tailed Datasetscited by 9
  32. 2025Synthetic Text Generation for Training Large Language Models via Gradient Matchingcited by 9
  33. 2023Distill Gold from Massive Ores: Bi-level Data Pruning towards Efficient Dataset Distillationcited by 8
  34. 2023Self-Supervised Dataset Distillation for Transfer Learningcited by 8
  35. 2023Fair Wasserstein Coresetscited by 8
  36. 2025DD-Ranking: Rethinking the Evaluation of Dataset Distillationcited by 8
  37. 2023Dataset Distillation Meets Provable Subset Selectioncited by 7
  38. 2023Dataset Distillation in Latent Spacecited by 7

50 most cited of 111 All 111 (JSON)