# Dynamic Multimodal Information Bottleneck for Multimodality Classification

arXiv: https://arxiv.org/abs/2311.01066 · HTML: https://mlsanity.com/paper/2311.01066/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 54 references that are in the map.
- First posted on arXiv 2 Nov 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Scalable and accurate deep learning for electronic health records](https://mlsanity.com/paper/1801.07860/) (2018)
- [Learning Robust Representations via Multi-View Information Bottleneck](https://mlsanity.com/paper/2002.07017/) (2020)
- [Wasserstein Dependency Measure for Representation Learning](https://mlsanity.com/paper/1903.11780/) (2019)
- [Variational Information Bottleneck for Effective Low-Resource Fine-Tuning](https://mlsanity.com/paper/2106.05469/) (2021)
- [Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification](https://mlsanity.com/paper/2104.02862/) (2021)
- [Combining 3D Image and Tabular Data via the Dynamic Affine Feature Map Transform](https://mlsanity.com/paper/2107.05990/) (2021)
- [Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review](https://mlsanity.com/paper/2203.15588/) (2022)
- [Multi-modal Graph Learning for Disease Prediction](https://mlsanity.com/paper/2203.05880/) (2022)
- [Temporal Feature Alignment and Mutual Information Maximization for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2203.15227/) (2022)
- [Variational Knowledge Distillation for Disease Classification in Chest X-Rays](https://mlsanity.com/paper/2103.10825/) (2021)
- [Indication as Prior Knowledge for Multimodal Disease Classification in Chest Radiographs with Transformers](https://mlsanity.com/paper/2202.06076/) (2022)
- [U-GAT: Multimodal Graph Attention Network for COVID-19 Outcome Prediction](https://mlsanity.com/paper/2108.00860/) (2021)

## Built on by (most cited first, 14 of 14)

- [A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models](https://mlsanity.com/paper/2502.17516/) (2025)
- [Learning Optimal Multimodal Information Bottleneck Representations](https://mlsanity.com/paper/2505.19996/) (2025)
- [Multimodal Data Integration for Precision Oncology: Challenges and Future Directions](https://mlsanity.com/paper/2406.19611/) (2024)
- [CRAFT: Adapting VLA Models to Contact-rich Manipulation via Force-aware Curriculum Fine-tuning](https://mlsanity.com/paper/2602.12532/) (2026)
- [BottleHumor: Self-Informed Humor Explanation using the Information Bottleneck Principle](https://mlsanity.com/paper/2502.18331/) (2025)
- [Multimodal Conditional Information Bottleneck for Generalizable AI-Generated Image Detection](https://mlsanity.com/paper/2505.15217/) (2025)
- [Towards an Incremental Unified Multimodal Anomaly Detection: Augmenting Multimodal Denoising From an Information Bottleneck Perspective](https://mlsanity.com/paper/2603.02629/) (2026)
- [CLIMD: A Curriculum Learning Framework for Imbalanced Multimodal Diagnosis](https://mlsanity.com/paper/2508.01594/) (2025)
- [Learning Task-Specific Sampling Strategy for Sparse-View CT Reconstruction](https://mlsanity.com/paper/2409.01544/) (2024)
- [The Describe-Then-Generate Bottleneck: How VLM Descriptions Alter Image Generation Outcomes](https://mlsanity.com/paper/2509.18179/) (2025)
- [Fuzzy Attention-based Border Rendering Network for Lung Organ Segmentation](https://mlsanity.com/paper/2406.16189/) (2024)
- [BTW: A Non-Parametric Variance Stabilization Framework for Multimodal Model Integration](https://mlsanity.com/paper/2508.18551/) (2025)
- [SMILE: Self-Explainable Multimodal Information Bottleneck for Medical Diagnosis](https://mlsanity.com/paper/2609.05174/) (2026)
- [SafeMol: Dual-Modality Safety Alignment for Molecular Multimodal Models](https://mlsanity.com/paper/2609.33640/) (2026)

## Abstract (arXiv)

Effectively leveraging multimodal data such as various images, laboratory tests and clinical information is gaining traction in a variety of AI-based medical diagnosis and prognosis tasks. Most existing multi-modal techniques only focus on enhancing their performance by leveraging the differences or shared features from various modalities and fusing feature across different modalities. These approaches are generally not optimal for clinical settings, which pose the additional challenges of limited training data, as well as being rife with redundant data or noisy modality channels, leading to subpar performance. To address this gap, we study the robustness of existing methods to data redundancy and noise and propose a generalized dynamic multimodal information bottleneck framework for attaining a robust fused feature representation. Specifically, our information bottleneck module serves to filter out the task-irrelevant information and noises in the fused feature, and we further introduce a sufficiency loss to prevent dropping of task-relevant information, thus explicitly preserving the sufficiency of prediction information in the distilled feature. We validate our model on an in-house and a public COVID19 dataset for mortality prediction as well as two public biomedical datasets for diagnostic tasks. Extensive experiments show that our method surpasses the state-of-the-art and is significantly more robust, being the only method to remain performance when large-scale noisy channels exist. Our code is publicly available at https://github.com/ayanglab/DMIB.