# TACFN: Transformer-based Adaptive Cross-modal Fusion Network for Multimodal Emotion Recognition

arXiv: https://arxiv.org/abs/2505.06536 · HTML: https://mlsanity.com/paper/2505.06536/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 40 references that are in the map.
- First posted on arXiv 10 May 2025.

## Builds on (most cited first, 19 of 19)

- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2102.04830/) (2021)
- [Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities](https://mlsanity.com/paper/1812.07809/) (2018)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors](https://mlsanity.com/paper/1811.09362/) (2018)
- [Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research](https://mlsanity.com/paper/2005.00357/) (2020)
- [Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysis](https://mlsanity.com/paper/1609.05244/) (2016)
- [Learn to Combine Modalities in Multimodal Deep Learning](https://mlsanity.com/paper/1805.11730/) (2018)
- [Speech Emotion Recognition with Dual-Sequence LSTM Architecture](https://mlsanity.com/paper/1910.08874/) (2019)
- [MSAF: Multimodal Split Attention Fusion](https://mlsanity.com/paper/2012.07175/) (2020)
- [Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition](https://mlsanity.com/paper/2009.09629/) (2020)
- [Low Rank Fusion based Transformers for Multimodal Sequences](https://mlsanity.com/paper/2007.02038/) (2020)
- [Multimodal Fusion with Deep Neural Networks for Audio-Video Emotion Recognition](https://mlsanity.com/paper/1907.03196/) (2019)

## Built on by (most cited first, 3 of 3)

- [CogniAlign: Word-Level Multimodal Speech Alignment with Gated Cross-Attention for Alzheimer's Detection](https://mlsanity.com/paper/2506.01890/) (2025)
- [Multimodal Video Emotion Recognition with Reliable Reasoning Priors](https://mlsanity.com/paper/2508.03722/) (2025)
- [Evidential Multimodal Transformers with Dempster-Shafer Fusion for Emotion Recognition](https://mlsanity.com/paper/2610.13013/) (2026)

## Abstract (arXiv)

The fusion technique is the key to the multimodal emotion recognition task. Recently, cross-modal attention-based fusion methods have demonstrated high performance and strong robustness. However, cross-modal attention suffers from redundant features and does not capture complementary features well. We find that it is not necessary to use the entire information of one modality to reinforce the other during cross-modal interaction, and the features that can reinforce a modality may contain only a part of it. To this end, we design an innovative Transformer-based Adaptive Cross-modal Fusion Network (TACFN). Specifically, for the redundant features, we make one modality perform intra-modal feature selection through a self-attention mechanism, so that the selected features can adaptively and efficiently interact with another modality. To better capture the complementary information between the modalities, we obtain the fused weight vector by splicing and use the weight vector to achieve feature reinforcement of the modalities. We apply TCAFN to the RAVDESS and IEMOCAP datasets. For fair comparison, we use the same unimodal representations to validate the effectiveness of the proposed fusion method. The experimental results show that TACFN brings a significant performance improvement compared to other methods and reaches the state-of-the-art. All code and models could be accessed from https://github.com/shuzihuaiyu/TACFN.