mlsanityData to 7 Oct 2026

Learning to Generate Synthetic Training Data using Gradient Matching and Implicit Differentiation

arXiv 2203.08559 ↗First posted 16 Mar 2022cs.LG cs.AI cs.NEcited by 11

Not on the map. Why?

Abstract (arXiv)

Using huge training datasets can be costly and inconvenient. This article explores various data distillation techniques that can reduce the amount of data required to successfully train deep networks. Inspired by recent ideas, we suggest new data distillation techniques based on generative teaching networks, gradient matching, and the Implicit Function Theorem. Experiments with the MNIST image classification problem show that the new methods are computationally more efficient than previous ones and allow to increase the performance of models trained on distilled data.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 13 of its 20 references that are in the map. First posted on arXiv 16 Mar 2022.

Builds on

13 of 20 references.

References · 13

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2016Deep Networks with Stochastic Depthcited by 973
  3. 2020Dataset Condensation with Gradient Matchingcited by 510
  4. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  5. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  6. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  7. 2018Dataset Distillationcited by 224
  8. 2020Distilled One-Shot Federated Learningcited by 108
  9. 2019Soft-Label Dataset Distillation and Text Dataset Distillationcited by 106
  10. 2019Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Datacited by 103
  11. 2019Generalized Inner Loop Meta-Learningcited by 92
  12. 2014Automatic Differentiation of Algorithms for Machine Learningcited by 7
1 more
  1. 2020New Properties of the Data Distillation Method When Working With Tabular Datacited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 3; 2026: 2.

11 papers.

  1. 2023M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancycited by 49
  2. 2025Dataset Distillation with Neural Characteristic Function: A Minmax Perspectivecited by 45
  3. 2024DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillationcited by 20
  4. 2025Hyperbolic Dataset Distillationcited by 15
  5. 2024Not All Samples Should Be Utilized Equally: Towards Understanding and Improving Dataset Distillationcited by 14
  6. 2024DD-RobustBench: An Adversarial Robustness Benchmark for Dataset Distillationcited by 8
  7. 2025Decoupled Audio-Visual Dataset Distillationcited by 6
  8. 2024Mitigating Bias in Dataset Distillationcited by 4
  9. 2023AST: Effective Dataset Distillation through Alignment with Smooth and High-Quality Expert Trajectoriescited by 1
  10. 2026Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Modelscited by 1
  11. 2026Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matchingcited by 0