# Multimodal Fusion with Deep Neural Networks for Audio-Video Emotion Recognition

arXiv: https://arxiv.org/abs/1907.03196 · HTML: https://mlsanity.com/paper/1907.03196/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2019.

## Built on by (most cited first, 12 of 12)

- [Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors](https://mlsanity.com/paper/2109.02993/) (2021)
- [MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition](https://mlsanity.com/paper/2305.07214/) (2023)
- [A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition](https://mlsanity.com/paper/2111.02172/) (2021)
- [Deep Equilibrium Multimodal Fusion](https://mlsanity.com/paper/2306.16645/) (2023)
- [We Care: Multimodal Depression Detection and Knowledge Infused Mental Health Therapeutic Response Generation](https://mlsanity.com/paper/2406.10561/) (2024)
- [TACFN: Transformer-based Adaptive Cross-modal Fusion Network for Multimodal Emotion Recognition](https://mlsanity.com/paper/2505.06536/) (2025)
- [Explainable Multimodal Regression via Information Decomposition](https://mlsanity.com/paper/2512.22102/) (2025)
- [Self-Augmented Multi-Modal Feature Embedding](https://mlsanity.com/paper/2103.04731/) (2021)
- [Towards Contrastive Learning in Music Video Domain](https://mlsanity.com/paper/2309.00347/) (2023)
- [Towards Privacy-Preserving Affect Recognition: A Two-Level Deep Learning Architecture](https://mlsanity.com/paper/2111.07344/) (2021)
- [Dynamic Modality and View Selection for Multimodal Emotion Recognition with Missing Modalities](https://mlsanity.com/paper/2404.12251/) (2024)
- [MVRS: The Multimodal Virtual Reality Stimuli-based Emotion Recognition Dataset](https://mlsanity.com/paper/2509.05330/) (2025)

## Abstract (arXiv)

This paper presents a novel deep neural network (DNN) for multimodal fusion of audio, video and text modalities for emotion recognition. The proposed DNN architecture has independent and shared layers which aim to learn the representation for each modality, as well as the best combined representation to achieve the best prediction. Experimental results on the AVEC Sentiment Analysis in the Wild dataset indicate that the proposed DNN can achieve a higher level of Concordance Correlation Coefficient (CCC) than other state-of-the-art systems that perform early fusion of modalities at feature-level (i.e., concatenation) and late fusion at score-level (i.e., weighted average) fusion. The proposed DNN has achieved CCCs of 0.606, 0.534, and 0.170 on the development partition of the dataset for predicting arousal, valence and liking, respectively.