# Multi-modal Conditional Attention Fusion for Dimensional Emotion Prediction

arXiv: https://arxiv.org/abs/1709.02251 · HTML: https://mlsanity.com/paper/1709.02251/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2017.

## Built on by (most cited first, 7 of 7)

- [Emotion Recognition with Pre-Trained Transformers Using Multimodal Signals](https://mlsanity.com/paper/2212.13885/) (2022)
- [Recursive Joint Attention for Audio-Visual Fusion in Regression based Emotion Recognition](https://mlsanity.com/paper/2304.07958/) (2023)
- [Multimodal Fusion Using Deep Learning Applied to Driver's Referencing of Outside-Vehicle Objects](https://mlsanity.com/paper/2107.12167/) (2021)
- [Data and models for stance and premise detection in COVID-19 tweets: insights from the Social Media Mining for Health (SMM4H) 2022 shared task](https://mlsanity.com/paper/2311.08057/) (2023)
- [A MultiModal Social Robot Toward Personalized Emotion Interaction](https://mlsanity.com/paper/2110.05186/) (2021)
- [Multimodal Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the Vehicle](https://mlsanity.com/paper/2202.07360/) (2022)
- [Revisiting Pre-training in Audio-Visual Learning](https://mlsanity.com/paper/2302.03533/) (2023)

## Abstract (arXiv)

Continuous dimensional emotion prediction is a challenging task where the fusion of various modalities usually achieves state-of-the-art performance such as early fusion or late fusion. In this paper, we propose a novel multi-modal fusion strategy named conditional attention fusion, which can dynamically pay attention to different modalities at each time step. Long-short term memory recurrent neural networks (LSTM-RNN) is applied as the basic uni-modality model to capture long time dependencies. The weights assigned to different modalities are automatically decided by the current input features and recent history information rather than being fixed at any kinds of situation. Our experimental results on a benchmark dataset AVEC2015 show the effectiveness of our method which outperforms several common fusion strategies for valence prediction.