# Sum-Rate-Distortion Function for Indirect Multiterminal Source Coding in Federated Learning

arXiv: https://arxiv.org/abs/2101.08696 · HTML: https://mlsanity.com/paper/2101.08696/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 21 Jan 2021.

## Builds on (most cited first, 25 of 25)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming](https://mlsanity.com/paper/1309.5549/) (2013)
- [Convex Optimization: Algorithms and Complexity](https://mlsanity.com/paper/1405.4980/) (2014)
- [signSGD: Compressed Optimisation for Non-Convex Problems](https://mlsanity.com/paper/1802.04434/) (2018)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training](https://mlsanity.com/paper/1712.01887/) (2017)
- [Adaptive Federated Learning in Resource Constrained Edge Computing Systems](https://mlsanity.com/paper/1804.05271/) (2018)
- [Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge](https://mlsanity.com/paper/1804.08333/) (2018)
- [TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning](https://mlsanity.com/paper/1705.07878/) (2017)
- [Sparse Communication for Distributed Gradient Descent](https://mlsanity.com/paper/1704.05021/) (2017)
- [Compressing Neural Networks with the Hashing Trick](https://mlsanity.com/paper/1504.04788/) (2015)
- [Exploiting Cyclic Symmetry in Convolutional Neural Networks](https://mlsanity.com/paper/1602.02660/) (2016)
- [Group Sparse Regularization for Deep Neural Networks](https://mlsanity.com/paper/1607.00485/) (2016)
- [Accelerating Federated Learning via Momentum Gradient Descent](https://mlsanity.com/paper/1910.03197/) (2019)
- [Convergence of Update Aware Device Scheduling for Federated Learning at the Wireless Edge](https://mlsanity.com/paper/2001.10402/) (2020)
- [Age-Based Scheduling Policy for Federated Learning in Mobile Edge Networks](https://mlsanity.com/paper/1910.14648/) (2019)
- [Robust Federated Learning with Noisy Communication](https://mlsanity.com/paper/1911.00251/) (2019)
- [Variance-based Gradient Compression for Efficient Distributed Deep Learning](https://mlsanity.com/paper/1802.06058/) (2018)
- [Data-Importance Aware User Scheduling for Communication-Efficient Edge Machine Learning](https://mlsanity.com/paper/1910.02214/) (2019)
- [ASAGA: Asynchronous Parallel SAGA](https://mlsanity.com/paper/1606.04809/) (2016)
- [Memory Bounded Deep Convolutional Networks](https://mlsanity.com/paper/1412.1442/) (2014)

## Built on by (most cited first, 2 of 2)

- [Fundamental Limits of Communication Efficiency for Model Aggregation in Distributed Learning: A Rate-Distortion Approach](https://mlsanity.com/paper/2206.13984/) (2022)
- [Digital Over-the-Air Federated Learning in Multi-Antenna Systems](https://mlsanity.com/paper/2302.14648/) (2023)

## Abstract (arXiv)

One of the main focus in federated learning (FL) is the communication efficiency since a large number of participating edge devices send their updates to the edge server at each round of the model training. Existing works reconstruct each model update from edge devices and implicitly assume that the local model updates are independent over edge devices. In FL, however, the model update is an indirect multi-terminal source coding problem, also called as the CEO problem where each edge device cannot observe directly the gradient that is to be reconstructed at the decoder, but is rather provided only with a noisy version. The existing works do not leverage the redundancy in the information transmitted by different edges. This paper studies the rate region for the indirect multiterminal source coding problem in FL. The goal is to obtain the minimum achievable rate at a particular upper bound of gradient variance. We obtain the rate region for the quadratic vector Gaussian CEO problem under unbiased estimator and derive an explicit formula of the sum-rate-distortion function in the special case where gradient are identical over edge device and dimension. Finally, we analyse communication efficiency of convex Minibatched SGD and non-convex Minibatched SGD based on the sum-rate-distortion function, respectively.