MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis
Not on the map. Why?
Abstract (arXiv)
Multimodal sentiment analysis relies on language, visual, and acoustic cues, but utterance-level modality quality may vary due to occlusion, background noise, motion blur, or imperfect transcripts, causing conventional fusion to over-trust unreliable modalities. We propose MRUF, a reliability-aware fusion method that combines multi-granularity routing with uncertainty-aware calibration. MRUF summarizes sentiment-relevant representations, performs subspace- and modality-level routing, and supervises modality routing with leave-one-out error increases to estimate utterance-level modality importance. It further predicts modality-wise uncertainty and refines modality gates through inverse-variance reweighting, while modality-invariant contrastive alignment stabilizes the shared representation space. Experiments on CMU-MOSI and CMU-MOSEI under aligned and unaligned settings show consistent improvements over strong baselines, and mechanism analysis verifies that modalities with higher predicted uncertainty receive lower fusion weights.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 27 of its 42 references that are in the map. First posted on arXiv 12 Jul 2026.
Builds on
27 of 42 references.
References · 27
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
- 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,340
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,009
- 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 639
- 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 430
- 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 269
- 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 263
- 2017Gated Multimodal Units for Information Fusioncited by 246
- 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 180
- 2018Learning Factorized Multimodal Representationscited by 171
- 2018Memory Fusion Network for Multi-view Sequential Learningcited by 167
- 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
15 more
- 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
- 2023Decoupled Multimodal Distilling for Emotion Recognitioncited by 76
- 2022Sound Adversarial Audio-Visual Navigationcited by 46
- 2023Measuring Acoustics with Collaborative Multiple Agentscited by 21
- 2022Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysiscited by 20
- 2025DOPE: Dual Object Perception-Enhancement Network for Vision-and-Language Navigationcited by 13
- 2025Audio-Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigationcited by 13
- 2025Iterative Residual Cross-Attention Mechanism: An Integrated Approach for Audio-Visual Navigation Taskscited by 13
- 2023CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognitioncited by 12
- 2025DGFNet: End-to-End Audio-Visual Source Separation Based on Dynamic Gating Fusioncited by 12
- 2025Dynamic Multi-Target Fusion for Efficient Audio-Visual Navigationcited by 12
- 2024BSS-CFFMA: Cross-Domain Feature Fusion and Multi-Attention Speech Enhancement Network based on Self-Supervised Embeddingcited by 9
- 2024VNet: A GAN-based Multi-Tier Discriminator Network for Speech Synthesis Vocoderscited by 9
- 2025Modality-Invariant Bidirectional Temporal Representation Distillation Network for Missing Multimodal Sentiment Analysiscited by 7
- 2025FSDENet: A Frequency and Spatial Domains based Detail Enhancement Network for Remote Sensing Semantic Segmentationcited by 6
Built on by
Not cited yet.