# Is Cross-Attention Preferable to Self-Attention for Multi-Modal Emotion Recognition?

arXiv: https://arxiv.org/abs/2202.09263 · HTML: https://mlsanity.com/paper/2202.09263/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 18 Feb 2022.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [End-to-End Multimodal Emotion Recognition using Deep Neural Networks](https://mlsanity.com/paper/1704.08619/) (2017)
- [Multimodal Speech Emotion Recognition Using Audio and Text](https://mlsanity.com/paper/1810.04635/) (2018)
- [Speech Emotion Recognition Using Multi-hop Attention Mechanism](https://mlsanity.com/paper/1904.10788/) (2019)
- [Attentive Modality Hopping Mechanism for Speech Emotion Recognition](https://mlsanity.com/paper/1912.00846/) (2019)
- [Cross-Modal Knowledge Transfer via Inter-Modal Translation and Alignment for Affect Recognition](https://mlsanity.com/paper/2108.00809/) (2021)

## Built on by (most cited first, 16 of 16)

- [HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2401.05698/) (2024)
- [Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysis](https://mlsanity.com/paper/2208.07589/) (2022)
- [Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognition](https://mlsanity.com/paper/2311.11009/) (2023)
- [Continuous-Time Audiovisual Fusion with Recurrence vs. Attention for In-The-Wild Affect Recognition](https://mlsanity.com/paper/2203.13285/) (2022)
- [A Survey on Image-text Multimodal Models](https://mlsanity.com/paper/2309.15857/) (2023)
- [Comment-aided Video-Language Alignment via Contrastive Pre-training for Short-form Video Humor Detection](https://mlsanity.com/paper/2402.09055/) (2024)
- [Exploring Emotion Expression Recognition in Older Adults Interacting with a Virtual Coach](https://mlsanity.com/paper/2311.05567/) (2023)
- [Stock Movement Prediction with Multimodal Stable Fusion via Gated Cross-Attention Mechanism](https://mlsanity.com/paper/2406.06594/) (2024)
- [3M-TRANSFORMER: A Multi-Stage Multi-Stream Multimodal Transformer for Embodied Turn-Taking Prediction](https://mlsanity.com/paper/2310.14859/) (2023)
- [Leveraging Speech for Gesture Detection in Multimodal Communication](https://mlsanity.com/paper/2404.14952/) (2024)
- [Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition](https://mlsanity.com/paper/2305.13583/) (2023)
- [Interaction is all You Need? A Study of Robots Ability to Understand and Execute](https://mlsanity.com/paper/2311.07150/) (2023)
- [Cross-Attention Fusion of Visual and Geometric Features for Large Vocabulary Arabic Lipreading](https://mlsanity.com/paper/2402.11520/) (2024)
- [A Depression Detection Method Based on Multi-Modal Feature Fusion Using Cross-Attention](https://mlsanity.com/paper/2407.12825/) (2024)
- [IsoNet: Causal Analysis of Multimodal Transformers for Neuromuscular Gesture Classification](https://mlsanity.com/paper/2506.16744/) (2025)
- [MF-GCN: A Multi-Frequency Graph Convolutional Network for Tri-Modal Depression Detection Using Eye-Tracking, Facial, and Acoustic Features](https://mlsanity.com/paper/2511.15675/) (2025)

## Abstract (arXiv)

Humans express their emotions via facial expressions, voice intonation and word choices. To infer the nature of the underlying emotion, recognition models may use a single modality, such as vision, audio, and text, or a combination of modalities. Generally, models that fuse complementary information from multiple modalities outperform their uni-modal counterparts. However, a successful model that fuses modalities requires components that can effectively aggregate task-relevant information from each modality. As cross-modal attention is seen as an effective mechanism for multi-modal fusion, in this paper we quantify the gain that such a mechanism brings compared to the corresponding self-attention mechanism. To this end, we implement and compare a cross-attention and a self-attention model. In addition to attention, each model uses convolutional layers for local feature extraction and recurrent layers for global sequential modelling. We compare the models using different modality combinations for a 7-class emotion classification task using the IEMOCAP dataset. Experimental results indicate that albeit both models improve upon the state-of-the-art in terms of weighted and unweighted accuracy for tri- and bi-modal configurations, their performance is generally statistically comparable. The code to replicate the experiments is available at https://github.com/smartcameras/SelfCrossAttn