# Multi-modal Speech Emotion Recognition via Feature Distribution Adaptation Network

arXiv: https://arxiv.org/abs/2410.22023 · HTML: https://mlsanity.com/paper/2410.22023/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 35 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Builds on (most cited first, 7 of 7)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Deep Subdomain Adaptation Network for Image Classification](https://mlsanity.com/paper/2106.09388/) (2021)
- [Advances and Challenges in Meta-Learning: A Technical Review](https://mlsanity.com/paper/2307.04722/) (2023)
- [Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors](https://mlsanity.com/paper/2205.10279/) (2022)
- [A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2310.20494/) (2023)
- [GM-TCNet: Gated Multi-scale Temporal Convolutional Network using Emotion Causality for Speech Emotion Recognition](https://mlsanity.com/paper/2210.15834/) (2022)

## Abstract (arXiv)

In this paper, we propose a novel deep inductive transfer learning framework, named feature distribution adaptation network, to tackle the challenging multi-modal speech emotion recognition problem. Our method aims to use deep transfer learning strategies to align visual and audio feature distributions to obtain consistent representation of emotion, thereby improving the performance of speech emotion recognition. In our model, the pre-trained ResNet-34 is utilized for feature extraction for facial expression images and acoustic Mel spectrograms, respectively. Then, the cross-attention mechanism is introduced to model the intrinsic similarity relationships of multi-modal features. Finally, the multi-modal feature distribution adaptation is performed efficiently with feed-forward network, which is extended using the local maximum mean discrepancy loss. Experiments are carried out on two benchmark datasets, and the results demonstrate that our model can achieve excellent performance compared with existing ones.