# Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents

arXiv: https://arxiv.org/abs/2312.01537 · HTML: https://mlsanity.com/paper/2312.01537/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 3 Dec 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023)
- [Soft-Label Dataset Distillation and Text Dataset Distillation](https://mlsanity.com/paper/1910.02551/) (2019)
- [Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data](https://mlsanity.com/paper/1912.07768/) (2019)
- [Bayesian Coreset Construction via Greedy Iterative Geodesic Ascent](https://mlsanity.com/paper/1802.01737/) (2018)
- [FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2207.09653/) (2022)
- [Input Sparsity Time Low-Rank Approximation via Ridge Leverage Score Sampling](https://mlsanity.com/paper/1511.07263/) (2015)
- [Training Gaussian Mixture Models at Scale via Coresets](https://mlsanity.com/paper/1703.08110/) (2017)
- [FedSynth: Gradient Compression via Synthetic Data in Federated Learning](https://mlsanity.com/paper/2204.01273/) (2022)
- [Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity](https://mlsanity.com/paper/2105.00562/) (2021)
- [FLIS: Clustered Federated Learning via Inference Similarity for Non-IID Data Distribution](https://mlsanity.com/paper/2208.09754/) (2022)
- [When Do Curricula Work in Federated Learning?](https://mlsanity.com/paper/2212.12712/) (2022)
- [Federated Learning on Virtual Heterogeneous Data with Local-global Distillation](https://mlsanity.com/paper/2303.02278/) (2023)

## Built on by (most cited first, 16 of 16)

- [The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions](https://mlsanity.com/paper/2502.05673/) (2025)
- [Group Distributionally Robust Dataset Distillation with Risk Minimization](https://mlsanity.com/paper/2402.04676/) (2024)
- [Diversity-Driven Generative Dataset Distillation Based on Diffusion Model with Self-Adaptive Memory](https://mlsanity.com/paper/2505.19469/) (2025)
- [Task-Specific Generative Dataset Distillation with Difficulty-Guided Sampling](https://mlsanity.com/paper/2507.03331/) (2025)
- [Tracing Back the Malicious Clients in Poisoning Attacks to Federated Learning](https://mlsanity.com/paper/2407.07221/) (2024)
- [Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation](https://mlsanity.com/paper/2512.00308/) (2025)
- [ATOM: Attention Mixer for Efficient Dataset Distillation](https://mlsanity.com/paper/2405.01373/) (2024)
- [ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models](https://mlsanity.com/paper/2406.12044/) (2024)
- [Dataset Distillation from First Principles: Integrating Core Information Extraction and Purposeful Learning](https://mlsanity.com/paper/2409.01410/) (2024)
- [MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning](https://mlsanity.com/paper/2409.06067/) (2024)
- [Dataset Distillation-based Hybrid Federated Learning on Non-IID Data](https://mlsanity.com/paper/2409.17517/) (2024)
- [FedWSIDD: Federated Whole Slide Image Classification via Dataset Distillation](https://mlsanity.com/paper/2506.15365/) (2025)
- [Gradient Compression May Hurt Generalization: A Remedy by Synthetic Data Guided Sharpness Aware Minimization](https://mlsanity.com/paper/2602.11584/) (2026)
- [Foreground-Aware Dataset Distillation via Dynamic Patch Selection](https://mlsanity.com/paper/2601.02727/) (2026)
- [Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching](https://mlsanity.com/paper/2602.05391/) (2026)
- [Federated Distillation for Whole Slide Image via Gaussian-Mixture Feature Alignment and Curriculum Integration](https://mlsanity.com/paper/2605.00578/) (2026)

## Abstract (arXiv)

Data heterogeneity presents significant challenges for federated learning (FL). Recently, dataset distillation techniques have been introduced, and performed at the client level, to attempt to mitigate some of these challenges. In this paper, we propose a highly efficient FL dataset distillation framework on the server side, significantly reducing both the computational and communication demands on local devices while enhancing the clients' privacy. Unlike previous strategies that perform dataset distillation on local devices and upload synthetic data to the server, our technique enables the server to leverage prior knowledge from pre-trained deep generative models to synthesize essential data representations from a heterogeneous model architecture. This process allows local devices to train smaller surrogate models while enabling the training of a larger global model on the server, effectively minimizing resource utilization. We substantiate our claim with a theoretical analysis, demonstrating the asymptotic resemblance of the process to the hypothetical ideal of completely centralized training on a heterogeneous dataset. Empirical evidence from our comprehensive experiments indicates our method's superiority, delivering an accuracy enhancement of up to 40% over non-dataset-distillation techniques in highly heterogeneous FL contexts, and surpassing existing dataset-distillation methods by 18%. In addition to the high accuracy, our framework converges faster than the baselines because rather than the server trains on several sets of heterogeneous data distributions, it trains on a multi-modal distribution. Our code is available at https://github.com/FedDG23/FedDG-main.git