# Deep Fusion: An Attention Guided Factorized Bilinear Pooling for Audio-video Emotion Recognition

arXiv: https://arxiv.org/abs/1901.04889 · HTML: https://mlsanity.com/paper/1901.04889/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jan 2019.

## Built on by (most cited first, 6 of 6)

- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition](https://mlsanity.com/paper/2111.02172/) (2021)
- [Affective Video Content Analysis: Decade Review and New Perspectives](https://mlsanity.com/paper/2310.17212/) (2023)
- [Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention](https://mlsanity.com/paper/2103.02362/) (2021)
- [A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learning](https://mlsanity.com/paper/2412.00049/) (2024)
- [Recent Advances and Challenges in Deep Audio-Visual Correlation Learning](https://mlsanity.com/paper/2202.13673/) (2022)

## Abstract (arXiv)

Automatic emotion recognition (AER) is a challenging task due to the abstract concept and multiple expressions of emotion. Although there is no consensus on a definition, human emotional states usually can be apperceived by auditory and visual systems. Inspired by this cognitive process in human beings, it's natural to simultaneously utilize audio and visual information in AER. However, most traditional fusion approaches only build a linear paradigm, such as feature concatenation and multi-system fusion, which hardly captures complex association between audio and video. In this paper, we introduce factorized bilinear pooling (FBP) to deeply integrate the features of audio and video. Specifically, the features are selected through the embedded attention mechanism from respective modalities to obtain the emotion-related regions. The whole pipeline can be completed in a neural network. Validated on the AFEW database of the audio-video sub-challenge in EmotiW2018, the proposed approach achieves an accuracy of 62.48%, outperforming the state-of-the-art result.