mlsanityData to 7 Oct 2026

TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis

arXiv 2009.02902 ↗First posted 7 Sep 2020cs.CLcited by 40

Not on the map. Why?

Abstract (arXiv)

Multimodal sentiment analysis is an important research area that predicts speaker's sentiment tendency through features extracted from textual, visual and acoustic modalities. The central challenge is the fusion method of the multimodal information. A variety of fusion methods have been proposed, but few of them adopt end-to-end translation models to mine the subtle correlation between modalities. Enlightened by recent success of Transformer in the area of machine translation, we propose a new fusion method, TransModality, to address the task of multimodal sentiment analysis. We assume that translation between modalities contributes to a better joint representation of speaker's utterance. With Transformer, the learned features embody the information both from the source modality and the target modality. We validate our model on multiple multimodal datasets: CMU-MOSI, MELD, IEMOCAP. The experiments show that our proposed method achieves the state-of-the-art performance.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 9; 2023: 4; 2024: 12; 2025: 8; 2026: 3.

40 papers.

Sort:

By area:

  1. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  2. 2022Cross-modal Prototype Driven Network for Radiology Report Generationcited by 58
  3. 2022CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimationcited by 39
  4. 2022Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalitiescited by 28
  5. 2023Multimodal Multi-loss Fusion Network for Sentiment Analysiscited by 26
  6. 2024IC3M: In-Car Multimodal Multi-object Monitoring for Abnormal Status of Both Driver and Passengerscited by 20
  7. 2022Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysiscited by 18
  8. 2021CAMul: Calibrated and Accurate Multi-view Time-Series Forecastingcited by 17
  9. 2024WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledgecited by 16
  10. 2024Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learningcited by 16
  11. 2023Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detectioncited by 13
  12. 2021ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptorscited by 12
28 more
  1. 2024A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognitioncited by 12
  2. 2024Recent Advances in Multimodal Affective Computing: An NLP Perspectivecited by 12
  3. 2024Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selectioncited by 10
  4. 2025Dynamic Uncertainty-aware Multimodal Fusion for Outdoor Health Monitoringcited by 10
  5. 2022MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequencescited by 8
  6. 2024Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detectioncited by 8
  7. 2022Cross-Modality Gated Attention Fusion for Multimodal Sentiment Analysiscited by 4
  8. 2024Toward Robust Multimodal Learning using Multimodal Foundational Modelscited by 4
  9. 2024Global Contrastive Training for Multimodal Electronic Health Records with Language Supervisioncited by 4
  10. 2024Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approachcited by 4
  11. 2025Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shiftscited by 4
  12. 2026CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learningcited by 4
  13. 2022A Multimodal Sensor Fusion Framework Robust to Missing Modalities for Person Recognitioncited by 3
  14. 2024Temporal Cross-Attention for Dynamic Embedding and Tokenization of Multimodal Electronic Health Recordscited by 3
  15. 2021Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attentioncited by 2
  16. 2021Survey of Generative Methods for Social Media Analysiscited by 1
  17. 2022Multimodal Transformer for Parallel Concatenated Variational Autoencoderscited by 1
  18. 2023Image Matters: A New Dataset and Empirical Study for Multimodal Hyperbole Detectioncited by 1
  19. 2024Exemplar Masking for Multimodal Incremental Learningcited by 1
  20. 2025Uncertainty-Aware Collaborative System of Large and Small Models for Multimodal Sentiment Analysiscited by 1
  21. 2025Calibrating Multimodal Consensus for Emotion Recognitioncited by 1
  22. 2026Benchmarking Multimodal Large Language Models for Missing Modality Completion in Product Cataloguescited by 1
  23. 2022QNet: A Quantum-native Sequence Encoder Architecturecited by 0
  24. 2025ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completioncited by 0
  25. 2025FSRF: Factorization-guided Semantic Recovery for Incomplete Multimodal Sentiment Analysiscited by 0
  26. 2025Buffer replay enhances the robustness of multimodal learning under missing-modalitycited by 0
  27. 2025A unified framework for detecting point and collective anomalies in operating system logs via collaborative transformerscited by 0
  28. 2026SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Datacited by 0