mlsanityData to 14 Oct 2026

Dataset Distillation with Convexified Implicit Gradients

arXiv 2302.06755 ↗First posted 13 Feb 2023cs.LG cs.CV stat.MLcited by 46

Not on the map. Why?

Abstract (arXiv)

We propose a new dataset distillation algorithm using reparameterization and convexification of implicit gradients (RCIG), that substantially improves the state-of-the-art. To this end, we first formulate dataset distillation as a bi-level optimization problem. Then, we show how implicit gradients can be effectively used to compute meta-gradient updates. We further equip the algorithm with a convexified approximation that corresponds to learning on top of a frozen finite-width neural tangent kernel. Finally, we improve bias in implicit gradients by parameterizing the neural network to enable analytical computation of final-layer parameters given the body parameters. RCIG establishes the new state-of-the-art on a diverse series of dataset distillation tasks. Notably, with one image per class, on resized ImageNet, RCIG sees on average a 108% improvement over the previous state-of-the-art distillation algorithm. Similarly, we observed a 66% gain over SOTA on Tiny-ImageNet and 37% on CIFAR-100.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 61 of its 89 references that are in the map. First posted on arXiv 13 Feb 2023.

Builds on

61 of 89 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  7. 2013Intriguing properties of neural networkscited by 5,155
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  9. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  10. 2016Deep Learning with Differential Privacycited by 2,728
  11. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  12. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
38 more
  1. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  3. 2018On Lazy Training in Differentiable Programmingcited by 675
  4. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  5. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  6. 2019Meta-Learning with Implicit Gradientscited by 525
  7. 2020Dataset Condensation with Gradient Matchingcited by 516
  8. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
  9. 2021Dataset Condensation with Distribution Matchingcited by 365
  10. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  11. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 291
  12. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  13. 2016Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagationcited by 259
  14. 2020Influence Functions in Deep Learning Are Fragilecited by 252
  15. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  16. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
  17. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 192
  18. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  19. 2018Machine Learning with Membership Privacy using Adversarial Regularizationcited by 186
  20. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  21. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  22. 2022If Influence Functions are the Answer, Then What is the Question?cited by 144
  23. 2022Liquid Structural State-Space Modelscited by 136
  24. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 120
  25. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  26. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  27. 2019Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functionscited by 105
  28. 2019Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Datacited by 104
  29. 2020Learning Long-Term Dependencies in Irregularly-Sampled Time Seriescited by 83
  30. 2016Coresets for Scalable Bayesian Logistic Regressioncited by 79
  31. 2018On Coresets for Logistic Regressioncited by 67
  32. 2021Closed-form Continuous-time Neural Modelscited by 65
  33. 2022Adaptive Second Order Coresets for Data-efficient Machine Learningcited by 60
  34. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  35. 2020Coresets for Near-Convex Functionscited by 38
  36. 2022Sample Condensation in Online Continual Learningcited by 31
  37. 2022On Implicit Bias in Overparameterized Bilevel Optimizationcited by 30
  38. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 15; 2025: 14; 2026: 5.

46 papers.

Sort:

By area:

  1. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  2. 2023Efficient Dataset Distillation via Minimax Diffusioncited by 83
  3. 2023M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancycited by 49
  4. 2023Position: AI Evaluation Should Learn from How We Test Humanscited by 36
  5. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 27
  6. 2024Dataset Distillation by Automatic Training Trajectoriescited by 26
  7. 2023You Only Condense Once: Two Rules for Pruning Condensed Datasetscited by 20
  8. 2024Multisize Dataset Condensationcited by 19
  9. 2025Dataset Distillation via Vision-Language Category Prototypecited by 19
  10. 2024Graph Data Condensation via Self-expressive Graph Structure Reconstructioncited by 18
  11. 2024Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarioscited by 18
  12. 2024Low-Rank Similarity Mining for Multimodal Dataset Distillationcited by 17
34 more
  1. 2024DANCE: Dual-View Distribution Alignment for Dataset Condensationcited by 15
  2. 2023Rethinking Backdoor Attacks on Dataset Distillation: A Kernel Method Perspectivecited by 10
  3. 2024Group Distributionally Robust Dataset Distillation with Risk Minimizationcited by 10
  4. 2024Dataset Condensation for Time Series Classification via Dual Domain Matchingcited by 9
  5. 2023Self-Supervised Dataset Distillation for Transfer Learningcited by 8
  6. 2025Training-Free Dataset Pruning for Instance Segmentationcited by 7
  7. 2023Adaptive Instrument Design for Indirect Experimentscited by 6
  8. 2024One Category One Prompt: Dataset Distillation using Diffusion Modelscited by 6
  9. 2025Unifying Dataset Pruning and Distillation for Efficient Large-scale Compressioncited by 6
  10. 2023Embarassingly Simple Dataset Distillationcited by 5
  11. 2024Label-Augmented Dataset Distillationcited by 5
  12. 2025Beyond Random: Automatic Inner-loop Optimization in Dataset Distillationcited by 5
  13. 2024Mitigating Bias in Dataset Distillationcited by 4
  14. 2024Prioritize Alignment in Dataset Distillationcited by 4
  15. 2025Improving Noise Efficiency in Privacy-preserving Dataset Distillationcited by 4
  16. 2023On the Size and Approximation Error of Distilled Setscited by 3
  17. 2024Distilling Datasets Into Less Than One Imagecited by 3
  18. 2024ATOM: Attention Mixer for Efficient Dataset Distillationcited by 3
  19. 2025CONCORD: Concept-Informed Diffusion for Dataset Distillationcited by 3
  20. 2025CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encodercited by 3
  21. 2026Multimodal Distribution Matching for Vision-Language Dataset Distillationcited by 3
  22. 2025DICE: Data Influence Cascade in Decentralized Learningcited by 2
  23. 2025Dataset Distillation as Data Compression: A Rate-Utility Perspectivecited by 2
  24. 2023Distilled Pruning: Using Synthetic Data to Win the Lotterycited by 1
  25. 2025Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximationcited by 1
  26. 2025A Discrepancy-Based Perspective on Dataset Condensationcited by 1
  27. 2026DIVER: Diving Deeper into Distilled Data via Expressive Semantic Recoverycited by 1
  28. 2024DRUPI: Dataset Reduction Using Privileged Informationcited by 0
  29. 2025Variational Bayesian Pseudo-Coresetcited by 0
  30. 2025Contrastive Learning-Enhanced Trajectory Matching for Small-Scale Dataset Distillationcited by 0
  31. 2025Distill Video Datasets into Imagescited by 0
  32. 2026Vector-Quantized Soft Label Compression for Dataset Distillationcited by 0
  33. 2026Federated Distillation for Whole Slide Image via Gaussian-Mixture Feature Alignment and Curriculum Integrationcited by 0
  34. 2026Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillationcited by 0