# TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis

arXiv: https://arxiv.org/abs/2009.02902 · HTML: https://mlsanity.com/paper/2009.02902/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2020.

## Built on by (most cited first, 30 of 40)

- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [Cross-modal Prototype Driven Network for Radiology Report Generation](https://mlsanity.com/paper/2207.04818/) (2022)
- [CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimation](https://mlsanity.com/paper/2207.14087/) (2022)
- [Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities](https://mlsanity.com/paper/2204.13707/) (2022)
- [Multimodal Multi-loss Fusion Network for Sentiment Analysis](https://mlsanity.com/paper/2308.00264/) (2023)
- [IC3M: In-Car Multimodal Multi-object Monitoring for Abnormal Status of Both Driver and Passengers](https://mlsanity.com/paper/2410.02592/) (2024)
- [Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis](https://mlsanity.com/paper/2207.11652/) (2022)
- [CAMul: Calibrated and Accurate Multi-view Time-Series Forecasting](https://mlsanity.com/paper/2109.07438/) (2021)
- [WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledge](https://mlsanity.com/paper/2401.06659/) (2024)
- [Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning](https://mlsanity.com/paper/2411.02793/) (2024)
- [Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection](https://mlsanity.com/paper/2306.02137/) (2023)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors](https://mlsanity.com/paper/2112.01368/) (2021)
- [A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2403.04245/) (2024)
- [Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selection](https://mlsanity.com/paper/2401.16685/) (2024)
- [Dynamic Uncertainty-aware Multimodal Fusion for Outdoor Health Monitoring](https://mlsanity.com/paper/2508.09085/) (2025)
- [MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences](https://mlsanity.com/paper/2210.12798/) (2022)
- [Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detection](https://mlsanity.com/paper/2402.05294/) (2024)
- [Cross-Modality Gated Attention Fusion for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2208.11893/) (2022)
- [Toward Robust Multimodal Learning using Multimodal Foundational Models](https://mlsanity.com/paper/2401.13697/) (2024)
- [Global Contrastive Training for Multimodal Electronic Health Records with Language Supervision](https://mlsanity.com/paper/2404.06723/) (2024)
- [Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach](https://mlsanity.com/paper/2408.07445/) (2024)
- [Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts](https://mlsanity.com/paper/2506.10452/) (2025)
- [CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning](https://mlsanity.com/paper/2602.04920/) (2026)
- [A Multimodal Sensor Fusion Framework Robust to Missing Modalities for Person Recognition](https://mlsanity.com/paper/2210.10972/) (2022)
- [Temporal Cross-Attention for Dynamic Embedding and Tokenization of Multimodal Electronic Health Records](https://mlsanity.com/paper/2403.04012/) (2024)
- [Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention](https://mlsanity.com/paper/2103.02362/) (2021)
- [Calibrating Multimodal Consensus for Emotion Recognition](https://mlsanity.com/paper/2510.20256/) (2025)
- [Survey of Generative Methods for Social Media Analysis](https://mlsanity.com/paper/2112.07041/) (2021)
- [Multimodal Transformer for Parallel Concatenated Variational Autoencoders](https://mlsanity.com/paper/2210.16174/) (2022)

## Abstract (arXiv)

Multimodal sentiment analysis is an important research area that predicts speaker's sentiment tendency through features extracted from textual, visual and acoustic modalities. The central challenge is the fusion method of the multimodal information. A variety of fusion methods have been proposed, but few of them adopt end-to-end translation models to mine the subtle correlation between modalities. Enlightened by recent success of Transformer in the area of machine translation, we propose a new fusion method, TransModality, to address the task of multimodal sentiment analysis. We assume that translation between modalities contributes to a better joint representation of speaker's utterance. With Transformer, the learned features embody the information both from the source modality and the target modality. We validate our model on multiple multimodal datasets: CMU-MOSI, MELD, IEMOCAP. The experiments show that our proposed method achieves the state-of-the-art performance.