# Exploring Emotion Features and Fusion Strategies for Audio-Video Emotion Recognition

arXiv: https://arxiv.org/abs/2012.13912 · HTML: https://mlsanity.com/paper/2012.13912/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Dec 2020.

## Built on by (most cited first, 24 of 24)

- [Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning](https://mlsanity.com/paper/2406.11161/) (2024)
- [The Effect of Model Compression on Fairness in Facial Expression Recognition](https://mlsanity.com/paper/2201.01709/) (2022)
- [A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition](https://mlsanity.com/paper/2111.02172/) (2021)
- [Learning from Synthetic Data: Facial Expression Classification based on Ensemble of Multi-task Networks](https://mlsanity.com/paper/2207.10025/) (2022)
- [Fuzzy Approach for Audio-Video Emotion Recognition in Computer Games for Children](https://mlsanity.com/paper/2309.00138/) (2023)
- [Hierarchical Audio-Visual Information Fusion with Multi-label Joint Decoding for MER 2023](https://mlsanity.com/paper/2309.07925/) (2023)
- [Self-supervised Adaptive Pre-training of Multilingual Speech Models for Language and Dialect Identification](https://mlsanity.com/paper/2312.07338/) (2023)
- [Facing Asymmetry -- Uncovering the Causal Link between Facial Symmetry and Expression Classifiers using Synthetic Interventions](https://mlsanity.com/paper/2409.15927/) (2024)
- [A Weakly Supervised Approach to Emotion-change Prediction and Improved Mood Inference](https://mlsanity.com/paper/2306.06979/) (2023)
- [Detail-Enhanced Intra- and Inter-modal Interaction for Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2405.16701/) (2024)
- [Signal Transformer: Complex-valued Attention and Meta-Learning for Signal Recognition](https://mlsanity.com/paper/2106.04392/) (2021)
- [Label quality in AffectNet: results of crowd-based re-annotation](https://mlsanity.com/paper/2110.04476/) (2021)
- [Event and Entity Extraction from Generated Video Captions](https://mlsanity.com/paper/2211.02982/) (2022)
- [Enhancing Multilingual Speech Recognition through Language Prompt Tuning and Frame-Level Language Adapter](https://mlsanity.com/paper/2309.09443/) (2023)
- [GMTR: Graph Matching Transformers](https://mlsanity.com/paper/2311.08141/) (2023)
- [DarkShot: Lighting Dark Images with Low-Compute and High-Quality](https://mlsanity.com/paper/2312.16805/) (2023)
- [TRLS: A Time Series Representation Learning Framework via Spectrogram for Medical Signal Processing](https://mlsanity.com/paper/2401.05431/) (2024)
- [TEN-GUARD: Tensor Decomposition for Backdoor Attack Detection in Deep Neural Networks](https://mlsanity.com/paper/2401.05432/) (2024)
- [T-Pixel2Mesh: Combining Global and Local Transformer for 3D Mesh Generation from a Single Image](https://mlsanity.com/paper/2403.13663/) (2024)
- [SZTU-CMU at MER2024: Improving Emotion-LLaMA with Conv-Attention for Multimodal Emotion Recognition](https://mlsanity.com/paper/2408.10500/) (2024)
- [Strategic Arms with Side Communication Prevail Over Low-Regret MAB Algorithms](https://mlsanity.com/paper/2408.17101/) (2024)
- [Towards General Deepfake Detection with Dynamic Curriculum](https://mlsanity.com/paper/2410.11162/) (2024)
- [Attr-Int: A Simple and Effective Entity Alignment Framework for Heterogeneous Knowledge Graphs](https://mlsanity.com/paper/2410.13409/) (2024)
- [Mask6D: Masked Pose Priors For 6D Object Pose Estimation](https://mlsanity.com/paper/2507.06486/) (2025)

## Abstract (arXiv)

The audio-video based emotion recognition aims to classify a given video into basic emotions. In this paper, we describe our approaches in EmotiW 2019, which mainly explores emotion features and feature fusion strategies for audio and visual modality. For emotion features, we explore audio feature with both speech-spectrogram and Log Mel-spectrogram and evaluate several facial features with different CNN models and different emotion pretrained strategies. For fusion strategies, we explore intra-modal and cross-modal fusion methods, such as designing attention mechanisms to highlights important emotion feature, exploring feature concatenation and factorized bilinear pooling (FBP) for cross-modal feature fusion. With careful evaluation, we obtain 65.5% on the AFEW validation set and 62.48% on the test set and rank third in the challenge.