# Towards efficient compression and communication for prototype-based decentralized learning

arXiv: https://arxiv.org/abs/2411.09267 · HTML: https://mlsanity.com/paper/2411.09267/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 36 references that are in the map.
- First posted on arXiv 14 Nov 2024.

## Builds on (most cited first, 12 of 12)

- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges](https://mlsanity.com/paper/2211.08413/) (2022)
- [Decentralized Federated Learning: A Survey and Perspective](https://mlsanity.com/paper/2306.01603/) (2023)
- [Distributed optimization over time-varying directed graphs](https://mlsanity.com/paper/1303.2289/) (2013)
- [The Representation Jensen-Shannon Divergence](https://mlsanity.com/paper/2305.16446/) (2023)
- [The Age of Gossip in Networks](https://mlsanity.com/paper/2102.02893/) (2021)
- [Prototype Helps Federated Learning: Towards Faster Convergence](https://mlsanity.com/paper/2303.12296/) (2023)
- [FedRFQ: Prototype-Based Federated Learning with Reduced Redundancy, Minimal Failure, and Enhanced Quality](https://mlsanity.com/paper/2401.07558/) (2024)
- [Boosting Federated Learning Convergence with Prototype Regularization](https://mlsanity.com/paper/2307.10575/) (2023)
- [Decentralised and collaborative machine learning framework for IoT](https://mlsanity.com/paper/2312.12190/) (2023)
- [Scale-Robust Timely Asynchronous Decentralized Learning](https://mlsanity.com/paper/2404.19749/) (2024)

## Abstract (arXiv)

In prototype-based federated learning, the exchange of model parameters between clients and the master server is replaced by transmission of prototypes or quantized versions of the data samples to the aggregation server. A fully decentralized deployment of prototype-based learning, without a central agregartor of prototypes, is more robust upon network failures and reacts faster to changes in the statistical distribution of the data, suggesting potential advantages and quick adaptation in dynamic learning tasks, e.g., when the data sources are IoT devices or when data is non-iid. In this paper, we consider the problem of designing a communication-efficient decentralized learning system based on prototypes. We address the challenge of prototype redundancy by leveraging on a twofold data compression technique, i.e., sending only update messages if the prototypes are informationtheoretically useful (via the Jensen-Shannon distance), and using clustering on the prototypes to compress the update messages used in the gossip protocol. We also use parallel instead of sequential gossiping, and present an analysis of its age-of-information (AoI). Our experimental results show that, with these improvements, the communications load can be substantially reduced without decreasing the convergence rate of the learning algorithm.