# Distributed Variational Representation Learning

arXiv: https://arxiv.org/abs/1807.04193 · HTML: https://mlsanity.com/paper/1807.04193/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jul 2018.

## Built on by (most cited first, 30 of 32)

- [Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications](https://mlsanity.com/paper/2207.09353/) (2022)
- [Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications](https://mlsanity.com/paper/2111.12444/) (2021)
- [Task-Oriented Communication for Multi-Device Cooperative Edge Inference](https://mlsanity.com/paper/2109.00172/) (2021)
- [Robust Information Bottleneck for Task-Oriented Communication with Digital Modulation](https://mlsanity.com/paper/2209.10382/) (2022)
- [Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learning](https://mlsanity.com/paper/2206.02604/) (2022)
- [Semantic Information Recovery in Wireless Networks](https://mlsanity.com/paper/2204.13366/) (2022)
- [Minimum Description Length and Generalization Guarantees for Representation Learning](https://mlsanity.com/paper/2402.03254/) (2024)
- [Information decomposition in complex systems via machine learning](https://mlsanity.com/paper/2307.04755/) (2023)
- [In-Network Learning: Distributed Training and Inference in Networks](https://mlsanity.com/paper/2107.03433/) (2021)
- [Semantics-Empowered Communication: A Tutorial-cum-Survey](https://mlsanity.com/paper/2212.08487/) (2022)
- [Lessons from Generalization Error Analysis of Federated Learning: You May Communicate Less Often!](https://mlsanity.com/paper/2306.05862/) (2023)
- [Interpretability with full complexity by constraining feature information](https://mlsanity.com/paper/2211.17264/) (2022)
- [Tackling Distribution Shifts in Task-Oriented Communication with Information Bottleneck](https://mlsanity.com/paper/2405.09514/) (2024)
- [Information flows of diverse autoencoders](https://mlsanity.com/paper/2102.07402/) (2021)
- [Multi-view Information Bottleneck Without Variational Approximation](https://mlsanity.com/paper/2204.10530/) (2022)
- [Semantic Edge Computing and Semantic Communications in 6G Networks: A Unifying Survey and Research Challenges](https://mlsanity.com/paper/2411.18199/) (2024)
- [Computationally Efficient Approximations for Matrix-based Renyi's Entropy](https://mlsanity.com/paper/2112.13720/) (2021)
- [Machine-learning optimized measurements of chaotic dynamical systems via the information bottleneck](https://mlsanity.com/paper/2311.04896/) (2023)
- [On In-network learning. A Comparative Study with Federated and Split Learning](https://mlsanity.com/paper/2104.14929/) (2021)
- [Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priors](https://mlsanity.com/paper/2502.15540/) (2025)
- [Adversarial Disentanglement by Backpropagation with Physics-Informed Variational Autoencoder](https://mlsanity.com/paper/2506.13658/) (2025)
- [Scalable Vector Gaussian Information Bottleneck](https://mlsanity.com/paper/2102.07525/) (2021)
- [Theoretical Analysis of Deep Neural Networks in Physical Layer Communication](https://mlsanity.com/paper/2202.09954/) (2022)
- [Elastic Information Bottleneck](https://mlsanity.com/paper/2311.03955/) (2023)
- [Toward Real-Time Edge AI: Model-Agnostic Task-Oriented Communication with Visual Feature Alignment](https://mlsanity.com/paper/2412.00862/) (2024)
- [Heterogeneity Matters even More in Distributed Learning: Study from Generalization Perspective](https://mlsanity.com/paper/2503.01598/) (2025)
- [Multiview Representation Learning via Distributed Joint Latent Space Structuring](https://mlsanity.com/paper/2504.18455/) (2025)
- [Distributed Information Bottleneck Theory for Multi-Modal Task-Aware Semantic Communication](https://mlsanity.com/paper/2510.04000/) (2025)
- [Characterizing information loss in a chaotic double pendulum with the Information Bottleneck](https://mlsanity.com/paper/2210.14220/) (2022)
- [Minimal Communication-Cost Statistical Learning](https://mlsanity.com/paper/2406.08193/) (2024)

## Abstract (arXiv)

The problem of distributed representation learning is one in which multiple sources of information X1,ldots,XK are processed separately so as to learn as much information as possible about some ground truth Y. We investigate this problem from information-theoretic grounds, through a generalization of Tishby's centralized Information Bottleneck (IB) method to the distributed setting. Specifically, K encoders, K ≥ 2, compress their observations X1,ldots,XK separately in a manner such that, collectively, the produced representations preserve as much information as possible about Y. We study both discrete memoryless (DM) and memoryless vector Gaussian data models. For the discrete model, we establish a single-letter characterization of the optimal tradeoff between complexity (or rate) and relevance (or information) for a class of memoryless sources (the observations X1,ldots,XK being conditionally independent given Y). For the vector Gaussian model, we provide an explicit characterization of the optimal complexity-relevance tradeoff. Furthermore, we develop a variational bound on the complexity-relevance tradeoff which generalizes the evidence lower bound (ELBO) to the distributed setting. We also provide two algorithms that allow to compute this bound: i) a Blahut-Arimoto type iterative algorithm which enables to compute optimal complexity-relevance encoding mappings by iterating over a set of self-consistent equations, and ii) a variational inference type algorithm in which the encoding mappings are parametrized by neural networks and the bound approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on synthetic and real datasets are provided to support the efficiency of the approaches and algorithms developed in this paper.