# Multilingual Speech Emotion Recognition With Multi-Gating Mechanism and Neural Architecture Search

arXiv: https://arxiv.org/abs/2211.08237 · HTML: https://mlsanity.com/paper/2211.08237/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 31 Oct 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Mish: A Self Regularized Non-Monotonic Activation Function](https://mlsanity.com/paper/1908.08681/) (2019)
- [Universal Phone Recognition with a Multilingual Allophone System](https://mlsanity.com/paper/2002.11800/) (2020)
- [SERAB: A multi-lingual benchmark for speech emotion recognition](https://mlsanity.com/paper/2110.03414/) (2021)
- [Fusing ASR Outputs in Joint Training for Speech Emotion Recognition](https://mlsanity.com/paper/2110.15684/) (2021)
- [CNN+LSTM Architecture for Speech Emotion Recognition with Data Augmentation](https://mlsanity.com/paper/1802.05630/) (2018)
- [Cross-lingual and Multilingual Speech Emotion Recognition on English and French](https://mlsanity.com/paper/1803.00357/) (2018)
- [An Introduction to Neural Architecture Search for Convolutional Networks](https://mlsanity.com/paper/2005.11074/) (2020)
- [Acoustics Based Intent Recognition Using Discovered Phonetic Units for Low Resource Languages](https://mlsanity.com/paper/2011.03646/) (2020)
- [Cross Lingual Cross Corpus Speech Emotion Recognition](https://mlsanity.com/paper/2003.07996/) (2020)

## Built on by (most cited first, 1 of 1)

- [Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognition](https://mlsanity.com/paper/2308.08713/) (2023)

## Abstract (arXiv)

Speech emotion recognition (SER) classifies audio into emotion categories such as Happy, Angry, Fear, Disgust and Neutral. While Speech Emotion Recognition (SER) is a common application for popular languages, it continues to be a problem for low-resourced languages, i.e., languages with no pretrained speech-to-text recognition models. This paper firstly proposes a language-specific model that extract emotional information from multiple pre-trained speech models, and then designs a multi-domain model that simultaneously performs SER for various languages. Our multidomain model employs a multi-gating mechanism to generate unique weighted feature combination for each language, and also searches for specific neural network structure for each language through a neural architecture search module. In addition, we introduce a contrastive auxiliary loss to build more separable representations for audio data. Our experiments show that our model raises the state-of-the-art accuracy by 3% for German and 14.3% for French.