# Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification

arXiv: https://arxiv.org/abs/2104.02862 · HTML: https://mlsanity.com/paper/2104.02862/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 7 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://mlsanity.com/paper/1903.07071/) (2019)
- [Learning Robust Representations via Multi-View Information Bottleneck](https://mlsanity.com/paper/2002.07017/) (2020)
- [Dynamic Dual-Attentive Aggregation Learning for Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2007.09314/) (2020)
- [Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow](https://mlsanity.com/paper/1810.00821/) (2018)
- [RGB-Infrared Cross-Modality Person Re-Identification via Joint Pixel and Feature Alignment](https://mlsanity.com/paper/1910.05839/) (2019)
- [Cross-modality Person re-identification with Shared-Specific Feature Transfer](https://mlsanity.com/paper/2002.12489/) (2020)
- [Ranked List Loss for Deep Metric Learning](https://mlsanity.com/paper/1903.03238/) (2019)
- [Cross-Modality Paired-Images Generation for RGB-Infrared Person Re-Identification](https://mlsanity.com/paper/2002.04114/) (2020)
- [Hi-CMD: Hierarchical Cross-Modality Disentanglement for Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/1912.01230/) (2019)
- [Efficient Estimation of Mutual Information for Strongly Dependent Variables](https://mlsanity.com/paper/1411.2003/) (2014)
- [Specializing Word Embeddings (for Parsing) by Information Bottleneck](https://mlsanity.com/paper/1910.00163/) (2019)
- [Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck](https://mlsanity.com/paper/1908.07094/) (2019)
- [The Dual Information Bottleneck](https://mlsanity.com/paper/2006.04641/) (2020)
- [Inverting Supervised Representations with Autoregressive Neural Density Models](https://mlsanity.com/paper/1806.00400/) (2018)
- [Finite-Sample Analysis of Fixed-k Nearest Neighbor Density Functional Estimators](https://mlsanity.com/paper/1606.01554/) (2016)

## Built on by (most cited first, 30 of 50)

- [Diverse Embedding Expansion Network and Low-Light Cross-Modality Benchmark for Visible-Infrared Person Re-identification](https://mlsanity.com/paper/2303.14481/) (2023)
- [Person Re-identification Method Based on Color Attack and Joint Defence](https://mlsanity.com/paper/2111.09571/) (2021)
- [Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application](https://mlsanity.com/paper/2407.01885/) (2024)
- [Exposing the Deception: Uncovering More Forgery Clues for Deepfake Detection](https://mlsanity.com/paper/2403.01786/) (2024)
- [Cross-Domain Correlation Distillation for Unsupervised Domain Adaptation in Nighttime Semantic Segmentation](https://mlsanity.com/paper/2205.00858/) (2022)
- [Shape-Erased Feature Learning for Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2304.04205/) (2023)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Learning Modal-Invariant and Temporal-Memory for Video-based Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2208.02450/) (2022)
- [Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information](https://mlsanity.com/paper/2401.08732/) (2024)
- [Temporal Feature Alignment and Mutual Information Maximization for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2203.15227/) (2022)
- [CLIP-Driven Semantic Discovery Network for Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2401.05806/) (2024)
- [Mutual Information-Based Temporal Difference Learning for Human Pose Estimation in Video](https://mlsanity.com/paper/2303.08475/) (2023)
- [CMTR: Cross-modality Transformer for Visible-infrared Person Re-identification](https://mlsanity.com/paper/2110.08994/) (2021)
- [Unsupervised Visible-Infrared Person ReID by Collaborative Learning with Neighbor-Guided Label Refinement](https://mlsanity.com/paper/2305.12711/) (2023)
- [Learning Optimal Multimodal Information Bottleneck Representations](https://mlsanity.com/paper/2505.19996/) (2025)
- [Dynamic Multimodal Information Bottleneck for Multimodality Classification](https://mlsanity.com/paper/2311.01066/) (2023)
- [Exploring Homogeneous and Heterogeneous Consistent Label Associations for Unsupervised Visible-Infrared Person ReID](https://mlsanity.com/paper/2402.00672/) (2024)
- [LongDanceDiff: Long-term Dance Generation with Conditional Diffusion Model](https://mlsanity.com/paper/2308.11945/) (2023)
- [Video-based Visible-Infrared Person Re-Identification with Auxiliary Samples](https://mlsanity.com/paper/2311.15571/) (2023)
- [Shape-centered Representation Learning for Visible-Infrared Person Re-identification](https://mlsanity.com/paper/2310.17952/) (2023)
- [Video-Level Language-Driven Video-Based Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2506.02439/) (2025)
- [MMD-ReID: A Simple but Effective Solution for Visible-Thermal Person ReID](https://mlsanity.com/paper/2111.05059/) (2021)
- [Bridging the Gap: Multi-Level Cross-Modality Joint Alignment for Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2307.08316/) (2023)
- [Exploring Invariant Representation for Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2302.00884/) (2023)
- [Few-shot Action Recognition via Intra- and Inter-Video Information Maximization](https://mlsanity.com/paper/2305.06114/) (2023)
- [Action Recognition with Multi-stream Motion Modeling and Mutual Information Maximization](https://mlsanity.com/paper/2306.07576/) (2023)
- [Semantic-Aligned Learning with Collaborative Refinement for Unsupervised VI-ReID](https://mlsanity.com/paper/2504.19244/) (2025)
- [Cross-Modality Earth Mover's Distance for Visible Thermal Person Re-Identification](https://mlsanity.com/paper/2203.01675/) (2022)
- [INSURE: An Information Theory Inspired Disentanglement and Purification Model for Domain Generalization](https://mlsanity.com/paper/2309.04063/) (2023)
- [Disentangled Noisy Correspondence Learning](https://mlsanity.com/paper/2408.05503/) (2024)

## Abstract (arXiv)

The Information Bottleneck (IB) provides an information theoretic principle for representation learning, by retaining all information relevant for predicting label while minimizing the redundancy. Though IB principle has been applied to a wide range of applications, its optimization remains a challenging problem which heavily relies on the accurate estimation of mutual information. In this paper, we present a new strategy, Variational Self-Distillation (VSD), which provides a scalable, flexible and analytic solution to essentially fitting the mutual information but without explicitly estimating it. Under rigorously theoretical guarantee, VSD enables the IB to grasp the intrinsic correlation between representation and label for supervised training. Furthermore, by extending VSD to multi-view learning, we introduce two other strategies, Variational Cross-Distillation (VCD) and Variational Mutual-Learning (VML), which significantly improve the robustness of representation to view-changes by eliminating view-specific and task-irrelevant information. To verify our theoretically grounded strategies, we apply our approaches to cross-modal person Re-ID, and conduct extensive experiments, where the superior performance against state-of-the-art methods are demonstrated. Our intriguing findings highlight the need to rethink the way to estimate mutual