# An Information-Theoretic Analysis of The Cost of Decentralization for Learning and Inference Under Privacy Constraints

arXiv: https://arxiv.org/abs/2110.05014 · HTML: https://mlsanity.com/paper/2110.05014/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 11 Oct 2021.

## Builds on (most cited first, 13 of 13)

- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [SecureBoost: A Lossless Federated Learning Framework](https://mlsanity.com/paper/1901.08755/) (2019)
- [Feature Inference Attack on Model Predictions in Vertical Federated Learning](https://mlsanity.com/paper/2010.10152/) (2020)
- [Differential Privacy as a Mutual Information Constraint](https://mlsanity.com/paper/1608.03677/) (2016)
- [Minimum Excess Risk in Bayesian Learning](https://mlsanity.com/paper/2012.14868/) (2020)
- [PyVertical: A Vertical Federated Learning Framework for Multi-headed SplitNN](https://mlsanity.com/paper/2104.00489/) (2021)
- [Information-Theoretic Bounds on the Generalization Error and Privacy Leakage in Federated Learning](https://mlsanity.com/paper/2005.02503/) (2020)
- [Privacy-Preserving Asynchronous Federated Learning Algorithms for Multi-Party Vertically Collaborative Learning](https://mlsanity.com/paper/2008.06233/) (2020)
- [A Vertical Federated Learning Method for Interpretable Scorecard and Its Application in Credit Scoring](https://mlsanity.com/paper/2009.06218/) (2020)
- [Rate-Distortion Analysis of Minimum Excess Risk in Bayesian Learning](https://mlsanity.com/paper/2105.04180/) (2021)
- [Federated Learning for Coalition Operations](https://mlsanity.com/paper/1910.06799/) (2019)

## Abstract (arXiv)

In vertical federated learning (FL), the features of a data sample are distributed across multiple agents. As such, inter-agent collaboration can be beneficial not only during the learning phase, as is the case for standard horizontal FL, but also during the inference phase. A fundamental theoretical question in this setting is how to quantify the cost, or performance loss, of decentralization for learning and/or inference. In this paper, we consider general supervised learning problems with any number of agents, and provide a novel information-theoretic quantification of the cost of decentralization in the presence of privacy constraints on inter-agent communication within a Bayesian framework. The cost of decentralization for learning and/or inference is shown to be quantified in terms of conditional mutual information terms involving features and label variables.