# DistDD: Distributed Data Distillation Aggregation through Gradient Matching

arXiv: https://arxiv.org/abs/2410.08665 · HTML: https://mlsanity.com/paper/2410.08665/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 26 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Builds on (most cited first, 22 of 22)

- [Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates](https://mlsanity.com/paper/1803.01498/) (2018)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Local Model Poisoning Attacks to Byzantine-Robust Federated Learning](https://mlsanity.com/paper/1911.11815/) (2019)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping](https://mlsanity.com/paper/2012.13995/) (2020)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation](https://mlsanity.com/paper/2211.11004/) (2022)
- [Improved Distribution Matching for Dataset Condensation](https://mlsanity.com/paper/2307.09742/) (2023)
- [Soft-Label Dataset Distillation and Text Dataset Distillation](https://mlsanity.com/paper/1910.02551/) (2019)
- [DENSE: Data-Free One-Shot Federated Learning](https://mlsanity.com/paper/2112.12371/) (2021)
- [Byzantine-Resilient Secure Federated Learning](https://mlsanity.com/paper/2007.11115/) (2020)
- [Flexible Dataset Distillation: Learn Labels Instead of Images](https://mlsanity.com/paper/2006.08572/) (2020)
- [FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2207.09653/) (2022)
- [Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environments](https://mlsanity.com/paper/2208.11311/) (2022)
- [Challenges and Approaches for Mitigating Byzantine Attacks in Federated Learning](https://mlsanity.com/paper/2112.14468/) (2021)
- [From Federated Learning to Federated Neural Architecture Search: A Survey](https://mlsanity.com/paper/2009.05868/) (2020)
- [Loss-Curvature Matching for Dataset Selection and Condensation](https://mlsanity.com/paper/2303.04449/) (2023)
- [DYNAFED: Tackling Client Data Heterogeneity with Global Dynamics](https://mlsanity.com/paper/2211.10878/) (2022)
- [Real-time Federated Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2003.02793/) (2020)
- [FLoRA: Single-shot Hyper-parameter Optimization for Federated Learning](https://mlsanity.com/paper/2112.08524/) (2021)
- [FedTune: Automatic Tuning of Federated Learning Hyper-Parameters from System Perspective](https://mlsanity.com/paper/2110.03061/) (2021)
- [Genetic CFL: Optimization of Hyper-Parameters in Clustered Federated Learning](https://mlsanity.com/paper/2107.07233/) (2021)

## Built on by (most cited first, 1 of 1)

- [The Composition Gap in Dataset Distillation](https://mlsanity.com/paper/2609.34343/) (2026)

## Abstract (arXiv)

In this paper, we introduce DistDD, a novel approach within the federated learning framework that reduces the need for repetitive communication by distilling data directly on clients' devices. Unlike traditional federated learning that requires iterative model updates across nodes, DistDD facilitates a one-time distillation process that extracts a global distilled dataset, maintaining the privacy standards of federated learning while significantly cutting down communication costs. By leveraging the DistDD's distilled dataset, the developers of the FL can achieve just-in-time parameter tuning and neural architecture search over FL without repeating the whole FL process multiple times. We provide a detailed convergence proof of the DistDD algorithm, reinforcing its mathematical stability and reliability for practical applications. Our experiments demonstrate the effectiveness and robustness of DistDD, particularly in non-i.i.d. and mislabeled data scenarios, showcasing its potential to handle complex real-world data challenges distinctively from conventional federated learning methods. We also evaluate DistDD's application in the use case and prove its effectiveness and communication-savings in the NAS use case.