# Multimodal Sentiment Analysis on CMU-MOSEI Dataset using Transformer-based Models

arXiv: https://arxiv.org/abs/2505.06110 · HTML: https://mlsanity.com/paper/2505.06110/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 19 references that are in the map.
- First posted on arXiv 9 May 2025.

## Builds on (most cited first, 5 of 5)

- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Joint Robust Voicing Detection and Pitch Estimation Based on Residual Harmonics](https://mlsanity.com/paper/2001.00459/) (2019)
- [Detection of Glottal Closure Instants from Speech Signals: a Quantitative Review](https://mlsanity.com/paper/2001.00473/) (2019)

## Abstract (arXiv)

This project performs multimodal sentiment analysis using the CMU-MOSEI dataset, using transformer-based models with early fusion to integrate text, audio, and visual modalities. We employ BERT-based encoders for each modality, extracting embeddings that are concatenated before classification. The model achieves strong performance, with 97.87% 7-class accuracy and a 0.9682 F1-score on the test set, demonstrating the effectiveness of early fusion in capturing cross-modal interactions. The training utilized Adam optimization (lr=1e-4), dropout (0.3), and early stopping to ensure generalization and robustness. Results highlight the superiority of transformer architectures in modeling multimodal sentiment, with a low MAE (0.1060) indicating precise sentiment intensity prediction. Future work may compare fusion strategies or enhance interpretability. This approach utilizes multimodal learning by effectively combining linguistic, acoustic, and visual cues for sentiment analysis.