# Variational Distillation for Multi-View Learning

arXiv: https://arxiv.org/abs/2206.09548 · HTML: https://mlsanity.com/paper/2206.09548/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 81 references that are in the map.
- First posted on arXiv 20 Jun 2022.

## Builds on (most cited first, 30 of 49)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds](https://mlsanity.com/paper/1911.11236/) (2019)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Deep Learning for Person Re-identification: A Survey and Outlook](https://mlsanity.com/paper/2001.04193/) (2020)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution](https://mlsanity.com/paper/1704.03915/) (2017)
- [Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution](https://mlsanity.com/paper/2007.16100/) (2020)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation](https://mlsanity.com/paper/2011.10033/) (2020)
- [PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation](https://mlsanity.com/paper/2003.14032/) (2020)
- [Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion](https://mlsanity.com/paper/2012.03762/) (2020)
- [Emergence of Invariance and Disentanglement in Deep Representations](https://mlsanity.com/paper/1706.01350/) (2017)
- [Learning Robust Representations via Multi-View Information Bottleneck](https://mlsanity.com/paper/2002.07017/) (2020)
- [EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection](https://mlsanity.com/paper/2007.08856/) (2020)
- [(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network](https://mlsanity.com/paper/2102.04530/) (2021)
- [On Deep Multi-View Representation Learning: Objectives and Optimization](https://mlsanity.com/paper/1602.01024/) (2016)
- [A Survey on Multi-view Learning](https://mlsanity.com/paper/1304.5634/) (2013)
- [Dynamic Dual-Attentive Aggregation Learning for Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2007.09314/) (2020)
- [Restricting the Flow: Information Bottlenecks for Attribution](https://mlsanity.com/paper/2001.00396/) (2020)
- [A Survey of Multi-View Representation Learning](https://mlsanity.com/paper/1610.01206/) (2016)
- [Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow](https://mlsanity.com/paper/1810.00821/) (2018)

## Built on by (most cited first, 3 of 3)

- [Tensor-based Graph Learning with Consistency and Specificity for Multi-view Clustering](https://mlsanity.com/paper/2403.18393/) (2024)
- [Dual-Space Modality Consistency Learning for Universal Cross-Modal Re-Identification](https://mlsanity.com/paper/2608.06943/) (2026)
- [When Semantically Consistent Encoding Meets View-Label Heterogeneity Modeling: A Unified Framework for Incomplete Multi-View Multi-Label Learning](https://mlsanity.com/paper/2609.07525/) (2026)

## Abstract (arXiv)

Information Bottleneck (IB) based multi-view learning provides an information theoretic principle for seeking shared information contained in heterogeneous data descriptions. However, its great success is generally attributed to estimate the multivariate mutual information which is intractable when the network becomes complicated. Moreover, the representation learning tradeoff, it i.e., prediction-compression and sufficiency-consistency tradeoff, makes the IB hard to satisfy both requirements simultaneously. In this paper, we design several variational information bottlenecks to exploit two key characteristics (it i.e., sufficiency and consistency) for multi-view representation learning. Specifically, we propose a Multi-View Variational Distillation (MV2D) strategy to provide a scalable, flexible and analytical solution to fitting MI by giving arbitrary input of viewpoints but without explicitly estimating it. Under rigorously theoretical guarantee, our approach enables IB to grasp the intrinsic correlation between observations and semantic labels, producing predictive and compact representations naturally. Also, our information-theoretic constraint can effectively neutralize the sensitivity to heterogeneous data by eliminating both task-irrelevant and view-specific information, preventing both tradeoffs in multiple view cases. To verify our theoretically grounded strategies, we apply our approaches to various benchmarks under three different applications. Extensive experiments to quantitatively and qualitatively demonstrate the effectiveness of our approach against state-of-the-art methods.