# Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis

arXiv: https://arxiv.org/abs/2310.05804 · HTML: https://mlsanity.com/paper/2310.05804/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2102.04830/) (2021)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021)
- [Expression Snippet Transformer for Robust Video-based Facial Expression Recognition](https://mlsanity.com/paper/2109.08409/) (2021)
- [M-SENA: An Integrated Platform for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2203.12441/) (2022)
- [SpeechFormer: A Hierarchical Efficient Framework Incorporating the Characteristics of Speech](https://mlsanity.com/paper/2203.03812/) (2022)
- [Noise-Resistant Multimodal Transformer for Emotion Recognition](https://mlsanity.com/paper/2305.02814/) (2023)

## Built on by (most cited first, 29 of 29)

- [WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledge](https://mlsanity.com/paper/2401.06659/) (2024)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark](https://mlsanity.com/paper/2504.16427/) (2025)
- [Tri-Subspaces Disentanglement for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2602.19585/) (2026)
- [CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning](https://mlsanity.com/paper/2602.04920/) (2026)
- [DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysis](https://mlsanity.com/paper/2504.11082/) (2025)
- [Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attention](https://mlsanity.com/paper/2410.15029/) (2024)
- [Disentangling Bias by Modeling Intra- and Inter-modal Causal Attention for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2508.04999/) (2025)
- [Towards Minimal Causal Representations for Human Multimodal Language Understanding](https://mlsanity.com/paper/2509.21805/) (2025)
- [PowMix: A Versatile Regularizer for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2312.12334/) (2023)
- [Robust Multimodal Sentiment Analysis via Double Information Bottleneck](https://mlsanity.com/paper/2511.01444/) (2025)
- [Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities](https://mlsanity.com/paper/2603.09111/) (2026)
- [Face-Guided Sentiment Boundary Enhancement for Weakly-Supervised Temporal Sentiment Localization](https://mlsanity.com/paper/2603.14750/) (2026)
- [Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis](https://mlsanity.com/paper/2604.12518/) (2026)
- [Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities](https://mlsanity.com/paper/2605.16889/) (2026)
- [Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective](https://mlsanity.com/paper/2608.30563/) (2026)
- [Multi-level Conflict-Aware Network for Multi-modal Sentiment Analysis](https://mlsanity.com/paper/2502.09675/) (2025)
- [SentiFormer: Metadata Enhanced Transformer for Image Sentiment Analysis](https://mlsanity.com/paper/2502.15322/) (2025)
- [MoLAN: A Unified Modality-Aware Noise Dynamic Editing Framework for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2508.09145/) (2025)
- [Semantic-Enhanced Cross-Modal Place Recognition for Robust Robot Localization](https://mlsanity.com/paper/2509.13474/) (2025)
- [QiNN-QJ: A Quantum-inspired Neural Network with Quantum Jump for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2510.27091/) (2025)
- [MIDG: Mixture of Invariant Experts with knowledge injection for Domain Generalization in Multimodal Sentiment Analysis](https://mlsanity.com/paper/2512.07430/) (2025)
- [Breaking the Correlation Plateau: On the Optimization and Capacity Limits of Attention-Based Regressors](https://mlsanity.com/paper/2602.17898/) (2026)
- [MIDAS: Mutual Information Disentanglement with Uncertainty-Aware Fusion for Incomplete Multimodal Sentiment Analysis](https://mlsanity.com/paper/2608.09986/) (2026)
- [Robust Incomplete Multimodal Sentiment Analysis via Iterative Proxy Correction](https://mlsanity.com/paper/2608.19971/) (2026)
- [SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data](https://mlsanity.com/paper/2609.30238/) (2026)
- [Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis](https://mlsanity.com/paper/2609.30470/) (2026)
- [Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness](https://mlsanity.com/paper/2610.04792/) (2026)
- [InstMoE: Adaptive Multimodal Routing with Specialized Experts](https://mlsanity.com/paper/2610.05111/) (2026)

## Abstract (arXiv)

Though Multimodal Sentiment Analysis (MSA) proves effective by utilizing rich information from multiple sources (e.g., language, video, and audio), the potential sentiment-irrelevant and conflicting information across modalities may hinder the performance from being further improved. To alleviate this, we present Adaptive Language-guided Multimodal Transformer (ALMT), which incorporates an Adaptive Hyper-modality Learning (AHL) module to learn an irrelevance/conflict-suppressing representation from visual and audio features under the guidance of language features at different scales. With the obtained hyper-modality representation, the model can obtain a complementary and joint representation through multimodal fusion for effective MSA. In practice, ALMT achieves state-of-the-art performance on several popular datasets (e.g., MOSI, MOSEI and CH-SIMS) and an abundance of ablation demonstrates the validity and necessity of our irrelevance/conflict suppression mechanism.