mlsanityData to 7 Oct 2026

Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis

arXiv 2109.00412 ↗First posted 1 Sep 2021cs.CL cs.AIcited by 130

Not on the map. Why?

Abstract (arXiv)

In multimodal sentiment analysis (MSA), the performance of a model highly depends on the quality of synthesized embeddings. These embeddings are generated from the upstream process called multimodal fusion, which aims to extract and combine the input unimodal raw data to produce a richer multimodal representation. Previous work either back-propagates the task loss or manipulates the geometric property of feature spaces to produce favorable fusion results, which neglects the preservation of critical task-related information that flows from input to the fusion results. In this work, we propose a framework named MultiModal InfoMax (MMIM), which hierarchically maximizes the Mutual Information (MI) in unimodal input pairs (inter-modality) and between multimodal fusion result and unimodal input in order to maintain task-related information through multimodal fusion. The framework is jointly trained with the main task (MSA) to improve the performance of the downstream MSA task. To address the intractable issue of MI bounds, we further formulate a set of computationally simple parametric and non-parametric methods to approximate their truth value. Experimental results on the two widely used datasets demonstrate the efficacy of our approach. The implementation of this work is publicly available at https://github.com/declare-lab/Multimodal-Infomax.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). Builds on 28 of its 49 references that are in the map. First posted on arXiv 1 Sep 2021.

Builds on

28 of 49 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  5. 2018Deep Graph Infomaxcited by 1,101
  6. 2016Deep Variational Information Bottleneckcited by 915
  7. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  8. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  9. 2018MINE: Mutual Information Neural Estimationcited by 680
  10. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  11. 2019On Variational Bounds of Mutual Informationcited by 550
  12. 2017Look, Listen and Learncited by 452
16 more
  1. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  2. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  3. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  4. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 233
  5. 2019DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversationcited by 184
  6. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 179
  7. 2018Learning Factorized Multimodal Representationscited by 171
  8. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  9. 2019Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysiscited by 82
  10. 2018Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principlecited by 70
  11. 2020Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Researchcited by 67
  12. 2019Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusioncited by 45
  13. 2018Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learningcited by 43
  14. 2019Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysiscited by 39
  15. 2016Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysiscited by 29
  16. 2018CASCADE: Contextual Sarcasm Detection in Online Discussion Forumscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 15; 2023: 21; 2024: 36; 2025: 26; 2026: 31.

130 papers.

Sort:

By area:

  1. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  2. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  3. 2022UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognitioncited by 61
  4. 2022Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Modulecited by 43
  5. 2024Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Predictioncited by 41
  6. 2023GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognitioncited by 38
  7. 2023Multimodal Variational Auto-encoder based Audio-Visual Segmentationcited by 37
  8. 2022Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Datacited by 36
  9. 2024Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalitiescited by 32
  10. 2023Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysiscited by 27
  11. 2023Multimodal Multi-loss Fusion Network for Sentiment Analysiscited by 26
  12. 2024Towards Multimodal Sentiment Analysis Debiasing via Bias Purificationcited by 26
38 more
  1. 2024Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognitioncited by 26
  2. 2024Towards Robust Multimodal Sentiment Analysis with Incomplete Datacited by 26
  3. 2024MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversationscited by 25
  4. 2022M-SENA: An Integrated Platform for Multimodal Sentiment Analysiscited by 21
  5. 2022Analyzing Modality Robustness in Multimodal Sentiment Analysiscited by 20
  6. 2022Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysiscited by 20
  7. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Taskscited by 18
  8. 2023Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognitioncited by 18
  9. 2024Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learningcited by 18
  10. 2024AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignmentcited by 17
  11. 2024WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledgecited by 16
  12. 2024Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learningcited by 16
  13. 2023Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognitioncited by 14
  14. 2023Revisiting Disentanglement and Fusion on Modality and Context in Conversational Multimodal Emotion Recognitioncited by 13
  15. 2022SANCL: Multimodal Review Helpfulness Prediction with Selective Attention and Natural Contrastive Learningcited by 12
  16. 2023Improving Multimodal Sentiment Analysis: Supervised Angular Margin-based Contrastive Learning for Enhanced Fusion Representationcited by 12
  17. 2024Recent Advances in Multimodal Affective Computing: An NLP Perspectivecited by 12
  18. 2023Enhanced Multimodal Representation Learning with Cross-modal KDcited by 11
  19. 2024Debiased Multimodal Understanding for Human Language Sequencescited by 10
  20. 2024Asynchronous Multimodal Video Sequence Fusion via Learning Modality-Exclusive and -Agnostic Representationscited by 10
  21. 2024DRIM: Learning Disentangled Representations from Incomplete Multimodal Healthcare Datacited by 10
  22. 2022MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequencescited by 8
  23. 2024UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Causecited by 8
  24. 2024Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Spacecited by 8
  25. 2024Tracing Intricate Cues in Dialogue: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion Recognitioncited by 8
  26. 2025Efficient Prompting for Continual Adaptation to Missing Modalitiescited by 8
  27. 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmarkcited by 8
  28. 2024Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysiscited by 7
  29. 2022Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysiscited by 6
  30. 2023TextMI: Textualize Multimodal Information for Integrating Non-verbal Cues in Pre-trained Language Modelscited by 6
  31. 2023VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learningcited by 6
  32. 2023Multimodal Prompt Transformer with Hybrid Contrastive Learning for Emotion Recognition in Conversationcited by 6
  33. 2025Improving Multimodal Sentiment Analysis via Modality Optimization and Dynamic Primary Modality Selectioncited by 6
  34. 2026AffectAgent: Collaborative Multi-Agent Reasoning for Retrieval-Augmented Multimodal Emotion Recognitioncited by 6
  35. 2024Semi-IIN: Semi-supervised Intra-inter modal Interaction Learning Network for Multimodal Sentiment Analysiscited by 5
  36. 2025Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphscited by 5
  37. 2026CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learningcited by 5
  38. 2023Denoising Bottleneck with Mutual Information Maximization for Video Multimodal Fusioncited by 4

50 most cited of 130 All 130 (JSON)