# Learning Optimal Multimodal Information Bottleneck Representations

arXiv: https://arxiv.org/abs/2505.19996 · HTML: https://mlsanity.com/paper/2505.19996/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 55 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [Learning Robust Representations via Multi-View Information Bottleneck](https://mlsanity.com/paper/2002.07017/) (2020)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Disease Prediction using Graph Convolutional Networks: Application to Autism Spectrum Disorder and Alzheimer's Disease](https://mlsanity.com/paper/1806.01738/) (2018)
- [Deep Learning-based Computational Pathology Predicts Origins for Cancers of Unknown Primary](https://mlsanity.com/paper/2006.13932/) (2020)
- [Convolutional Neural Networks at Constrained Time Cost](https://mlsanity.com/paper/1412.1710/) (2014)
- [Multimodal Representation Learning by Alternating Unimodal Adaptation](https://mlsanity.com/paper/2311.10707/) (2023)
- [Efficient Large-Scale Multi-Modal Classification](https://mlsanity.com/paper/1802.02892/) (2018)
- [Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification](https://mlsanity.com/paper/2104.02862/) (2021)
- [Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations](https://mlsanity.com/paper/2210.17444/) (2022)
- [The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation](https://mlsanity.com/paper/2206.06487/) (2022)
- [Majority Vote of Diverse Classifiers for Late Fusion](https://mlsanity.com/paper/1404.7796/) (2014)
- [Dynamic Multimodal Information Bottleneck for Multimodality Classification](https://mlsanity.com/paper/2311.01066/) (2023)
- [An Information Criterion for Controlled Disentanglement of Multimodal Data](https://mlsanity.com/paper/2410.23996/) (2024)
- [Enhancing Multimodal Entity and Relation Extraction with Variational Information Bottleneck](https://mlsanity.com/paper/2304.02328/) (2023)
- [Domain Adaptive and Fine-grained Anomaly Detection for Single-cell Sequencing Data and Beyond](https://mlsanity.com/paper/2404.17454/) (2024)
- [MEATRD: Multimodal Anomalous Tissue Region Detection Enhanced with Spatial Transcriptomics](https://mlsanity.com/paper/2412.10659/) (2024)
- [Dual Advancement of Representation Learning and Clustering for Sparse and Noisy Images](https://mlsanity.com/paper/2409.01781/) (2024)
- [A Methodological Framework for Measuring Spatial Labeling Similarity](https://mlsanity.com/paper/2505.14128/) (2025)

## Built on by (most cited first, 15 of 15)

- [Fusion or Confusion? Multimodal Complexity Is Not All You Need](https://mlsanity.com/paper/2512.22991/) (2025)
- [LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck](https://mlsanity.com/paper/2606.13061/) (2026)
- [A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models](https://mlsanity.com/paper/2603.29676/) (2026)
- [Disentangling Bias by Modeling Intra- and Inter-modal Causal Attention for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2508.04999/) (2025)
- [Towards Minimal Causal Representations for Human Multimodal Language Understanding](https://mlsanity.com/paper/2509.21805/) (2025)
- [Adapter-Augmented Bandits for Online Multi-Constrained Multi-Modal Inference Scheduling](https://mlsanity.com/paper/2603.06403/) (2026)
- [The Describe-Then-Generate Bottleneck: How VLM Descriptions Alter Image Generation Outcomes](https://mlsanity.com/paper/2509.18179/) (2025)
- [Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition](https://mlsanity.com/paper/2605.21565/) (2026)
- [OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment](https://mlsanity.com/paper/2605.29900/) (2026)
- [When Are Multimodal Predictions Biologically Supported? A Diagnostic Evaluation Framework](https://mlsanity.com/paper/2605.31504/) (2026)
- [RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities](https://mlsanity.com/paper/2606.05109/) (2026)
- [SMILE: Self-Explainable Multimodal Information Bottleneck for Medical Diagnosis](https://mlsanity.com/paper/2609.05174/) (2026)
- [Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis](https://mlsanity.com/paper/2609.30470/) (2026)
- [VaME: Exploring Variational Latent Reasoning for Multimodal Embeddings](https://mlsanity.com/paper/2609.33402/) (2026)
- [SafeMol: Dual-Modality Safety Alignment for Molecular Multimodal Models](https://mlsanity.com/paper/2609.33640/) (2026)

## Abstract (arXiv)

Leveraging high-quality joint representations from multimodal data can greatly enhance model performance in various machine-learning based applications. Recent multimodal learning methods, based on the multimodal information bottleneck (MIB) principle, aim to generate optimal MIB with maximal task-relevant information and minimal superfluous information via regularization. However, these methods often set ad hoc regularization weights and overlook imbalanced task-relevant information across modalities, limiting their ability to achieve optimal MIB. To address this gap, we propose a novel multimodal learning framework, Optimal Multimodal Information Bottleneck (OMIB), whose optimization objective guarantees the achievability of optimal MIB by setting the regularization weight within a theoretically derived bound. OMIB further addresses imbalanced task-relevant information by dynamically adjusting regularization weights per modality, promoting the inclusion of all task-relevant information. Moreover, we establish a solid information-theoretical foundation for OMIB's optimization and implement it under the variational approximation framework for computational efficiency. Finally, we empirically validate the OMIB's theoretical properties on synthetic data and demonstrate its superiority over the state-of-the-art benchmark methods in various downstream tasks.