# Learning to Generate Synthetic Training Data using Gradient Matching and Implicit Differentiation

arXiv: https://arxiv.org/abs/2203.08559 · HTML: https://mlsanity.com/paper/2203.08559/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 13 of 13)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Optimizing Millions of Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/1911.02590/) (2019)
- [Dataset Distillation](https://mlsanity.com/paper/1811.10959/) (2018)
- [Distilled One-Shot Federated Learning](https://mlsanity.com/paper/2009.07999/) (2020)
- [Soft-Label Dataset Distillation and Text Dataset Distillation](https://mlsanity.com/paper/1910.02551/) (2019)
- [Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data](https://mlsanity.com/paper/1912.07768/) (2019)
- [Generalized Inner Loop Meta-Learning](https://mlsanity.com/paper/1910.01727/) (2019)
- [Automatic Differentiation of Algorithms for Machine Learning](https://mlsanity.com/paper/1404.7456/) (2014)
- [New Properties of the Data Distillation Method When Working With Tabular Data](https://mlsanity.com/paper/2010.09839/) (2020)

## Built on by (most cited first, 11 of 11)

- [M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy](https://mlsanity.com/paper/2312.15927/) (2023)
- [Dataset Distillation with Neural Characteristic Function: A Minmax Perspective](https://mlsanity.com/paper/2502.20653/) (2025)
- [DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation](https://mlsanity.com/paper/2404.00264/) (2024)
- [Hyperbolic Dataset Distillation](https://mlsanity.com/paper/2505.24623/) (2025)
- [Not All Samples Should Be Utilized Equally: Towards Understanding and Improving Dataset Distillation](https://mlsanity.com/paper/2408.12483/) (2024)
- [DD-RobustBench: An Adversarial Robustness Benchmark for Dataset Distillation](https://mlsanity.com/paper/2403.13322/) (2024)
- [Decoupled Audio-Visual Dataset Distillation](https://mlsanity.com/paper/2511.17890/) (2025)
- [Mitigating Bias in Dataset Distillation](https://mlsanity.com/paper/2406.06609/) (2024)
- [Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models](https://mlsanity.com/paper/2605.07194/) (2026)
- [AST: Effective Dataset Distillation through Alignment with Smooth and High-Quality Expert Trajectories](https://mlsanity.com/paper/2310.10541/) (2023)
- [Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching](https://mlsanity.com/paper/2602.05391/) (2026)

## Abstract (arXiv)

Using huge training datasets can be costly and inconvenient. This article explores various data distillation techniques that can reduce the amount of data required to successfully train deep networks. Inspired by recent ideas, we suggest new data distillation techniques based on generative teaching networks, gradient matching, and the Implicit Function Theorem. Experiments with the MNIST image classification problem show that the new methods are computationally more efficient than previous ones and allow to increase the performance of models trained on distilled data.