mlsanityData to 14 Oct 2026

Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations

arXiv 2210.17444 ↗First posted 31 Oct 2022cs.LGcited by 48

Not on the map. Why?

Abstract (arXiv)

Learning effective joint embedding for cross-modal data has always been a focus in the field of multimodal machine learning. We argue that during multimodal fusion, the generated multimodal embedding may be redundant, and the discriminative unimodal information may be ignored, which often interferes with accurate prediction and leads to a higher risk of overfitting. Moreover, unimodal representations also contain noisy information that negatively influences the learning of cross-modal dynamics. To this end, we introduce the multimodal information bottleneck (MIB), aiming to learn a powerful and sufficient multimodal representation that is free of redundancy and to filter out noisy information in unimodal representations. Specifically, inheriting from the general information bottleneck (IB), MIB aims to learn the minimal sufficient representation for a given task by maximizing the mutual information between the representation and the target and simultaneously constraining the mutual information between the representation and the input data. Different from general IB, our MIB regularizes both the multimodal and unimodal representations, which is a comprehensive and flexible framework that is compatible with any fusion methods. We develop three MIB variants, namely, early-fusion MIB, late-fusion MIB, and complete MIB, to focus on different perspectives of information constraints. Experimental results suggest that the proposed method reaches state-of-the-art performance on the tasks of multimodal sentiment analysis and multimodal emotion recognition across three widely used datasets. The codes are available at urlhttps://github.com/TmacMai/Multimodal-Information-Bottleneck.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 31 of its 58 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

31 of 58 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  4. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,200
  5. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  6. 2016Deep Variational Information Bottleneckcited by 930
  7. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  8. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 433
  9. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 270
  10. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 265
  11. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  12. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 194
19 more
  1. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 191
  2. 2018Memory Fusion Network for Multi-view Sequential Learningcited by 168
  3. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  4. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 105
  5. 2018Multi-attention Recurrent Network for Human Communication Comprehensioncited by 89
  6. 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
  7. 2019InfoBot: Transfer and Exploration via the Information Bottleneckcited by 78
  8. 2019Learning Efficient Multi-agent Communication: An Information Bottleneck Approachcited by 61
  9. 2019Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusioncited by 45
  10. 2018Multimodal Language Analysis with Recurrent Multistage Fusioncited by 44
  11. 2021A Variational Information Bottleneck Approach to Multi-Omics Data Integrationcited by 40
  12. 2019Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysiscited by 39
  13. 2019Learning Representations from Imperfect Time Series Data via Tensor Rank Regularizationcited by 30
  14. 2020Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversationcited by 27
  15. 2018Investigating Audio, Visual, and Text Fusion Methods for End-to-End Automatic Personality Predictioncited by 22
  16. 2020Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognitioncited by 17
  17. 2019Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneckcited by 15
  18. 2021Quantum-inspired Multimodal Fusion for Video Sentiment Analysiscited by 13
  19. 2020Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling Fusioncited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 24; 2026: 13.

48 papers.

Sort:

By area:

  1. 2024Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Predictioncited by 41
  2. 2024Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learningcited by 18
  3. 2025Learning Optimal Multimodal Information Bottleneck Representationscited by 15
  4. 2023Speech-Text Dialog Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignmentcited by 13
  5. 2025Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descentcited by 13
  6. 2024Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Spacecited by 8
  7. 2023Survey of Action Recognition, Spotting and Spatio-Temporal Localization in Soccer -- Current Trends and Research Perspectivescited by 7
  8. 2025BottleHumor: Self-Informed Humor Explanation using the Information Bottleneck Principlecited by 6
  9. 2023Unleashing the Power of CNN and Transformer for Balanced RGB-Event Video Recognitioncited by 5
  10. 2025Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representationcited by 5
  11. 2025Multimodal Conditional Information Bottleneck for Generalizable AI-Generated Image Detectioncited by 5
  12. 2024Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognitioncited by 4
36 more
  1. 2025Satellite Edge Artificial Intelligence with Large Models: Architectures and Technologiescited by 4
  2. 2025A Multimodal-Multitask Framework with Cross-modal Relation and Hierarchical Interactive Attention for Semantic Comprehensioncited by 4
  3. 2025InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signalscited by 3
  4. 2025Towards Explainable Fusion and Balanced Learning in Multimodal Sentiment Analysiscited by 3
  5. 2025VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DRcited by 3
  6. 2025Multimodal Representation Learning and Fusioncited by 3
  7. 2025Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Promptingcited by 3
  8. 2024Cross-modality Guidance-aided Multi-modal Learning with Dual Attention for MRI Brain Tumor Gradingcited by 2
  9. 2024TimeSieve: Extracting Temporal Dynamics through Information Bottleneckscited by 2
  10. 2025Disentangling Bias by Modeling Intra- and Inter-modal Causal Attention for Multimodal Sentiment Analysiscited by 2
  11. 2025Towards Minimal Causal Representations for Human Multimodal Language Understandingcited by 2
  12. 2025Explainable Multimodal Regression via Information Decompositioncited by 2
  13. 2026CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal Fusioncited by 2
  14. 2025Uncertainty-Aware Collaborative System of Large and Small Models for Multimodal Sentiment Analysiscited by 1
  15. 2025Maximally Useful and Minimally Redundant: The Key to Self Supervised Learning for Imbalanced Datacited by 1
  16. 2025Distributed Information Bottleneck Theory for Multi-Modal Task-Aware Semantic Communicationcited by 1
  17. 2025Robust Multimodal Sentiment Analysis via Double Information Bottleneckcited by 1
  18. 2026MATEX: Multi-scale Attention and Text-guided Explainability of Medical Vision-Language Modelscited by 1
  19. 2026ModalImmune: Immunity Driven Unlearning via Self Destructive Trainingcited by 1
  20. 2026SeaVIS: Sound-Enhanced Association for Online Audio-Visual Instance Segmentationcited by 1
  21. 2026Addressing Missing and Noisy Modalities in One Solution: Unified Modality-Quality Framework for Low-quality Multimodal Datacited by 1
  22. 2026Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspectivecited by 1
  23. 2024Complementary Information Mutual Learning for Multimodality Medical Image Segmentationcited by 0
  24. 2024An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignmentcited by 0
  25. 2025Multi-Modal Artificial Intelligence of Embryo Grading and Pregnancy Prediction in Assisted Reproductive Technology: A Reviewcited by 0
  26. 2025MoLAN: A Unified Modality-Aware Noise Dynamic Editing Framework for Multimodal Sentiment Analysiscited by 0
  27. 2025Conditional Information Bottleneck for Multimodal Fusion: Overcoming Shortcut Learning in Sarcasm Detectioncited by 0
  28. 2025AIM: Adaptive Intra-Network Modulation for Balanced Multimodal Learningcited by 0
  29. 2025Multimodal Representation-disentangled Information Bottleneck for Multimodal Recommendationcited by 0
  30. 2026GRCF: Two-Stage Groupwise Ranking and Calibration Framework for Multimodal Sentiment Analysiscited by 0
  31. 2026QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysiscited by 0
  32. 2026A Conflict-Aware Penalty and Statistical Loss Framework for Balancing Modalities and Enhancing Stability in Multimodal Sentiment Analysiscited by 0
  33. 2026OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignmentcited by 0
  34. 2026SMILE: Self-Explainable Multimodal Information Bottleneck for Medical Diagnosiscited by 0
  35. 2026PRI-Net: A Lightweight Multimodal Framework for 3D UAV Localizationcited by 0
  36. 2026Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysiscited by 0