mlsanityData to 7 Oct 2026

Dataset Meta-Learning from Kernel Ridge-Regression

arXiv 2011.00050 ↗First posted 30 Oct 2020cs.LG stat.MLcited by 230

Not on the map. Why?

Abstract (arXiv)

One of the most fundamental aspects of any machine learning algorithm is the training data used by the algorithm. We introduce the novel concept of ε-approximation of datasets, obtaining datasets which are much smaller than or are significant corruptions of the original training data while maintaining similar model performance. We introduce a meta-learning algorithm called Kernel Inducing Points (KIP) for obtaining such remarkable datasets, inspired by the recent developments in the correspondence between infinitely-wide neural networks and kernel ridge-regression (KRR). For KRR tasks, we demonstrate that KIP can compress datasets by one or two orders of magnitude, significantly improving previous dataset distillation and subset selection methods while obtaining state of the art results for MNIST and CIFAR-10 classification. Furthermore, our KIP-learned datasets are transferable to the training of finite-width neural networks even beyond the lazy-training regime, which leads to state of the art results for neural network dataset distillation with potential applications to privacy-preservation.

Key facts. Cited by 230 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 35; 2023: 60; 2024: 62; 2025: 47; 2026: 22.

230 papers.

Sort:

By area:

  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
  3. 2021Dataset Condensation with Distribution Matchingcited by 362
  4. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  5. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  6. 2022Scaling Up Dataset Distillation to ImageNet-1K with Constant Memorycited by 180
  7. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  8. 2021Graph Condensation for Graph Neural Networkscited by 150
  9. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  10. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
  11. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  12. 2023Generalizing Dataset Distillation via Deep Generative Priorcited by 125
38 more
  1. 2023Improved Distribution Matching for Dataset Condensationcited by 124
  2. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 119
  3. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  4. 2022Dataset Distillation via Factorizationcited by 115
  5. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  6. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  7. 2022Dataset Condensation with Contrastive Signalscited by 105
  8. 2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
  9. 2022Condensing Graphs via One-Step Gradient Matchingcited by 100
  10. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 93
  11. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  12. 2023Data Distillation: A Surveycited by 86
  13. 2023DataDAM: Efficient Dataset Distillation with Attention Matchingcited by 83
  14. 2023Efficient Dataset Distillation via Minimax Diffusioncited by 81
  15. 2022Synthesizing Informative Training Samples with GANcited by 77
  16. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 76
  17. 2023Structure-free Graph Condensation: From Large-scale Graphs to Condensed Graph-free Datacited by 71
  18. 2023Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matchingcited by 69
  19. 2022DC-BENCH: Dataset Condensation Benchmarkcited by 67
  20. 2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 65
  21. 2023Large-scale Dataset Pruning with Dynamic Uncertaintycited by 63
  22. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  23. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  24. 2024A Comprehensive Survey on Graph Reduction: Sparsification, Coarsening, and Condensationcited by 57
  25. 2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
  26. 2023M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancycited by 49
  27. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  28. 2023DiM: Distilling Dataset into Generative Modelcited by 45
  29. 2023Does Graph Distillation See Like Vision Dataset Counterpart?cited by 45
  30. 2025Dataset Distillation with Neural Characteristic Function: A Minmax Perspectivecited by 45
  31. 2022Delving into Effective Gradient Matching for Dataset Condensationcited by 40
  32. 2023Sequential Subset Matching for Dataset Distillationcited by 40
  33. 2024Elucidating the Design Space of Dataset Condensationcited by 40
  34. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  35. 2023Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruningcited by 38
  36. 2022Submodularity In Machine Learning and Artificial Intelligencecited by 37
  37. 2023Fast Graph Condensation with Structure-based Neural Tangent Kernelcited by 36
  38. 2022Dataset Condensation with Latent Space Knowledge Factorization and Sharingcited by 34

50 most cited of 230 All 230 (JSON)