# Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side Distillation

arXiv: https://arxiv.org/abs/2401.14211 · HTML: https://mlsanity.com/paper/2401.14211/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 25 Jan 2024.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [Sparsified SGD with Memory](https://mlsanity.com/paper/1809.07599/) (2018)
- [Federated learning with hierarchical clustering of local updates to improve training on non-IID data](https://mlsanity.com/paper/2004.11791/) (2020)
- [FedKD: Communication Efficient Federated Learning via Knowledge Distillation](https://mlsanity.com/paper/2108.13323/) (2021)
- [Learning to See by Looking at Noise](https://mlsanity.com/paper/2106.05963/) (2021)
- [FEDZIP: A Compression Framework for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2102.01593/) (2021)
- [Federated Learning of Large Models at the Edge via Principal Sub-Model Training](https://mlsanity.com/paper/2208.13141/) (2022)
- [Slashing Communication Traffic in Federated Learning by Transmitting Clustered Model Updates](https://mlsanity.com/paper/2105.04153/) (2021)
- [Federated Progressive Sparsification (Purge, Merge, Tune)+](https://mlsanity.com/paper/2204.12430/) (2022)
- [The Augmented Image Prior: Distilling 1000 Classes by Extrapolating from a Single Image](https://mlsanity.com/paper/2112.00725/) (2021)
- [Comfetch: Federated Learning of Large Networks on Constrained Clients via Sketching](https://mlsanity.com/paper/2109.08346/) (2021)

## Built on by (most cited first, 4 of 4)

- [EncCluster: Scalable Functional Encryption in Federated Learning through Weight Clustering and Probabilistic Filters](https://mlsanity.com/paper/2406.09152/) (2024)
- [FedX: Explanation-Guided Pruning for Communication-Efficient Federated Learning in Remote Sensing](https://mlsanity.com/paper/2508.06256/) (2025)
- [Mitigating Noise Detriment in Differentially Private Federated Learning with Model Pre-training](https://mlsanity.com/paper/2408.09478/) (2024)
- [A Full Compression Pipeline for Green Federated Learning in Communication-Constrained Environments](https://mlsanity.com/paper/2604.11146/) (2026)

## Abstract (arXiv)

Federated Learning (FL) is a promising technique for the collaborative training of deep neural networks across multiple devices while preserving data privacy. Despite its potential benefits, FL is hindered by excessive communication costs due to repeated server-client communication during training. To address this challenge, model compression techniques, such as sparsification and weight clustering are applied, which often require modifying the underlying model aggregation schemes or involve cumbersome hyperparameter tuning, with the latter not only adjusts the model's compression rate but also limits model's potential for continuous improvement over growing data. In this paper, we propose FedCompress, a novel approach that combines dynamic weight clustering and server-side knowledge distillation to reduce communication costs while learning highly generalizable models. Through a comprehensive evaluation on diverse public datasets, we demonstrate the efficacy of our approach compared to baselines in terms of communication costs and inference speed.