mlsanityData to 7 Oct 2026

Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis

arXiv 2310.05804 ↗First posted 9 Oct 2023cs.AI cs.CL cs.CVcited by 27

Not on the map. Why?

Abstract (arXiv)

Though Multimodal Sentiment Analysis (MSA) proves effective by utilizing rich information from multiple sources (e.g., language, video, and audio), the potential sentiment-irrelevant and conflicting information across modalities may hinder the performance from being further improved. To alleviate this, we present Adaptive Language-guided Multimodal Transformer (ALMT), which incorporates an Adaptive Hyper-modality Learning (AHL) module to learn an irrelevance/conflict-suppressing representation from visual and audio features under the guidance of language features at different scales. With the obtained hyper-modality representation, the model can obtain a complementary and joint representation through multimodal fusion for effective MSA. In practice, ALMT achieves state-of-the-art performance on several popular datasets (e.g., MOSI, MOSEI and CH-SIMS) and an abundance of ablation demonstrates the validity and necessity of our irrelevance/conflict suppression mechanism.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 15 of its 29 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

15 of 29 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  6. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  7. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  8. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  9. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 179
  10. 2018Learning Factorized Multimodal Representationscited by 171
  11. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 130
  12. 2021Expression Snippet Transformer for Robust Video-based Facial Expression Recognitioncited by 33
3 more
  1. 2022M-SENA: An Integrated Platform for Multimodal Sentiment Analysiscited by 21
  2. 2022SpeechFormer: A Hierarchical Efficient Framework Incorporating the Characteristics of Speechcited by 20
  3. 2023Noise-Resistant Multimodal Transformer for Emotion Recognitioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 11; 2026: 12.

27 papers.

Sort:

By area:

  1. 2024WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledgecited by 16
  2. 2024Recent Advances in Multimodal Affective Computing: An NLP Perspectivecited by 12
  3. 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmarkcited by 8
  4. 2026Tri-Subspaces Disentanglement for Multimodal Sentiment Analysiscited by 5
  5. 2026CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learningcited by 4
  6. 2025DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysiscited by 3
  7. 2024Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attentioncited by 2
  8. 2025Disentangling Bias by Modeling Intra- and Inter-modal Causal Attention for Multimodal Sentiment Analysiscited by 2
  9. 2025Towards Minimal Causal Representations for Human Multimodal Language Understandingcited by 2
  10. 2023PowMix: A Versatile Regularizer for Multimodal Sentiment Analysiscited by 1
  11. 2025Robust Multimodal Sentiment Analysis via Double Information Bottleneckcited by 1
  12. 2026Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalitiescited by 1
15 more
  1. 2026Face-Guided Sentiment Boundary Enhancement for Weakly-Supervised Temporal Sentiment Localizationcited by 1
  2. 2026Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysiscited by 1
  3. 2026Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalitiescited by 1
  4. 2025Multi-level Conflict-Aware Network for Multi-modal Sentiment Analysiscited by 0
  5. 2025SentiFormer: Metadata Enhanced Transformer for Image Sentiment Analysiscited by 0
  6. 2025MoLAN: A Unified Modality-Aware Noise Dynamic Editing Framework for Multimodal Sentiment Analysiscited by 0
  7. 2025Semantic-Enhanced Cross-Modal Place Recognition for Robust Robot Localizationcited by 0
  8. 2025QiNN-QJ: A Quantum-inspired Neural Network with Quantum Jump for Multimodal Sentiment Analysiscited by 0
  9. 2025MIDG: Mixture of Invariant Experts with knowledge injection for Domain Generalization in Multimodal Sentiment Analysiscited by 0
  10. 2026Breaking the Correlation Plateau: On the Optimization and Capacity Limits of Attention-Based Regressorscited by 0
  11. 2026MIDAS: Mutual Information Disentanglement with Uncertainty-Aware Fusion for Incomplete Multimodal Sentiment Analysiscited by 0
  12. 2026Robust Incomplete Multimodal Sentiment Analysis via Iterative Proxy Correctioncited by 0
  13. 2026Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspectivecited by 0
  14. 2026SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Datacited by 0
  15. 2026Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysiscited by 0