# Distributed Estimation, Information Loss and Exponential Families

arXiv: https://arxiv.org/abs/1410.2653 · HTML: https://mlsanity.com/paper/1410.2653/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2014.

## Built on by (most cited first, 6 of 6)

- [High-Dimensional Distributed Sparse Classification with Scalable Communication-Efficient Global Updates](https://mlsanity.com/paper/2407.06346/) (2024)
- [Federated Averaging Langevin Dynamics: Toward a unified theory and new algorithms](https://mlsanity.com/paper/2211.00100/) (2022)
- [Byzantine-tolerant distributed learning of finite mixture models](https://mlsanity.com/paper/2407.13980/) (2024)
- [Fast proxy centers for Jeffreys centroids: The Jeffreys-Fisher-Rao and the inductive Gauss-Bregman centers](https://mlsanity.com/paper/2410.14326/) (2024)
- [Minimax and Communication-Efficient Distributed Best Subset Selection with Oracle Property](https://mlsanity.com/paper/2408.17276/) (2024)
- [Federated Soft Clustering via Generalized Total Variation Minimization](https://mlsanity.com/paper/2609.19202/) (2026)

## Abstract (arXiv)

Distributed learning of probabilistic models from multiple data repositories with minimum communication is increasingly important. We study a simple communication-efficient learning framework that first calculates the local maximum likelihood estimates (MLE) based on the data subsets, and then combines the local MLEs to achieve the best possible approximation to the global MLE given the whole dataset. We study this framework's statistical properties, showing that the efficiency loss compared to the global setting relates to how much the underlying distribution families deviate from full exponential families, drawing connection to the theory of information loss by Fisher, Rao and Efron. We show that the "full-exponential-family-ness" represents the lower bound of the error rate of arbitrary combinations of local MLEs, and is achieved by a KL-divergence-based combination method but not by a more common linear combination method. We also study the empirical properties of both methods, showing that the KL method significantly outperforms linear combination in practical settings with issues such as model misspecification, non-convexity, and heterogeneous data partitions.