# Meta Knowledge Condensation for Federated Learning

arXiv: https://arxiv.org/abs/2209.14851 · HTML: https://mlsanity.com/paper/2209.14851/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 29 Sep 2022.

## Builds on (most cited first, 29 of 29)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Personalized Federated Learning with Moreau Envelopes](https://mlsanity.com/paper/2006.08848/) (2020)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection](https://mlsanity.com/paper/1907.09693/) (2019)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Personalized Cross-Silo Federated Learning on Non-IID Data](https://mlsanity.com/paper/2007.03797/) (2020)
- [Dataset Distillation using Neural Feature Regression](https://mlsanity.com/paper/2206.00719/) (2022)
- [Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning](https://mlsanity.com/paper/2203.09249/) (2022)
- [On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms](https://mlsanity.com/paper/1908.10400/) (2019)
- [Distilled One-Shot Federated Learning](https://mlsanity.com/paper/2009.07999/) (2020)
- [DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training](https://mlsanity.com/paper/2206.00187/) (2022)
- [Federated Knowledge Distillation](https://mlsanity.com/paper/2011.02367/) (2020)
- [Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum](https://mlsanity.com/paper/2006.15815/) (2020)
- [The Advantage of Conditional Meta-Learning for Biased Regularization and Fine-Tuning](https://mlsanity.com/paper/2008.10857/) (2020)
- [Node Selection Toward Faster Convergence for Federated Learning on Non-IID Data](https://mlsanity.com/paper/2105.07066/) (2021)
- [Acceleration of Federated Learning with Alleviated Forgetting in Local Training](https://mlsanity.com/paper/2203.02645/) (2022)
- [Federated Few-Shot Learning with Adversarial Learning](https://mlsanity.com/paper/2104.00365/) (2021)
- [FedFOR: Stateless Heterogeneous Federated Learning with First-Order Regularization](https://mlsanity.com/paper/2209.10537/) (2022)

## Built on by (most cited first, 26 of 26)

- [Dataset Distillation: A Comprehensive Review](https://mlsanity.com/paper/2301.07014/) (2023)
- [A Comprehensive Survey of Dataset Distillation](https://mlsanity.com/paper/2301.05603/) (2023)
- [Data Distillation: A Survey](https://mlsanity.com/paper/2301.04272/) (2023)
- [Efficient Dataset Distillation via Minimax Diffusion](https://mlsanity.com/paper/2311.15529/) (2023)
- [Sequential Subset Matching for Dataset Distillation](https://mlsanity.com/paper/2311.01570/) (2023)
- [An Aggregation-Free Federated Learning for Tackling Data Heterogeneity](https://mlsanity.com/paper/2404.18962/) (2024)
- [The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions](https://mlsanity.com/paper/2502.05673/) (2025)
- [A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiency](https://mlsanity.com/paper/2307.10655/) (2023)
- [Importance-Aware Adaptive Dataset Distillation](https://mlsanity.com/paper/2401.15863/) (2024)
- [Not All Samples Should Be Utilized Equally: Towards Understanding and Improving Dataset Distillation](https://mlsanity.com/paper/2408.12483/) (2024)
- [TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendation](https://mlsanity.com/paper/2502.02854/) (2025)
- [Can pre-trained models assist in dataset distillation?](https://mlsanity.com/paper/2310.03295/) (2023)
- [One-shot Federated Learning via Synthetic Distiller-Distillate Communication](https://mlsanity.com/paper/2412.05186/) (2024)
- [Group Distributionally Robust Dataset Distillation with Risk Minimization](https://mlsanity.com/paper/2402.04676/) (2024)
- [DD-RobustBench: An Adversarial Robustness Benchmark for Dataset Distillation](https://mlsanity.com/paper/2403.13322/) (2024)
- [Subgraph Federated Learning for Local Generalization](https://mlsanity.com/paper/2503.03995/) (2025)
- [ATOM: Attention Mixer for Efficient Dataset Distillation](https://mlsanity.com/paper/2405.01373/) (2024)
- [Dataset Distillation-based Hybrid Federated Learning on Non-IID Data](https://mlsanity.com/paper/2409.17517/) (2024)
- [Tackling Data Heterogeneity in Federated Time Series Forecasting](https://mlsanity.com/paper/2411.15716/) (2024)
- [Exemplar-condensed Federated Class-incremental Learning](https://mlsanity.com/paper/2412.18926/) (2024)
- [AST: Effective Dataset Distillation through Alignment with Smooth and High-Quality Expert Trajectories](https://mlsanity.com/paper/2310.10541/) (2023)
- [Breaking Resource Barriers in Speech Emotion Recognition via Data Distillation](https://mlsanity.com/paper/2406.15119/) (2024)
- [One-Shot Collaborative Data Distillation](https://mlsanity.com/paper/2408.02266/) (2024)
- [High-Energy Concentration for Federated Learning in Frequency Domain](https://mlsanity.com/paper/2509.12630/) (2025)
- [Foreground-Aware Dataset Distillation via Dynamic Patch Selection](https://mlsanity.com/paper/2601.02727/) (2026)
- [CD^2: Constrained Dataset Distillation for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2601.08519/) (2026)

## Abstract (arXiv)

Existing federated learning paradigms usually extensively exchange distributed models at a central solver to achieve a more powerful model. However, this would incur severe communication burden between a server and multiple clients especially when data distributions are heterogeneous. As a result, current federated learning methods often require a large number of communication rounds in training. Unlike existing paradigms, we introduce an alternative perspective to significantly decrease the communication cost in federate learning. In this work, we first introduce a meta knowledge representation method that extracts meta knowledge from distributed clients. The extracted meta knowledge encodes essential information that can be used to improve the current model. As the training progresses, the contributions of training samples to a federated model also vary. Thus, we introduce a dynamic weight assignment mechanism that enables samples to contribute adaptively to the current model update. Then, informative meta knowledge from all active clients is sent to the server for model update. Training a model on the combined meta knowledge without exposing original data among different clients can significantly mitigate the heterogeneity issues. Moreover, to further ameliorate data heterogeneity, we also exchange meta knowledge among clients as conditional initialization for local meta knowledge extraction. Extensive experiments demonstrate the effectiveness and efficiency of our proposed method. Remarkably, our method outperforms the state-of-the-art by a large margin (from 74.07% to 92.95%) on MNIST with a restricted communication budget (i.e. 10 rounds).