# Towards adversarial learning of speaker-invariant representation for speech emotion recognition

arXiv: https://arxiv.org/abs/1903.09606 · HTML: https://mlsanity.com/paper/1903.09606/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Mar 2019.

## Built on by (most cited first, 4 of 4)

- [Denoising Induction Motor Sounds Using an Autoencoder](https://mlsanity.com/paper/2208.04462/) (2022)
- [DS-TDNN: Dual-stream Time-delay Neural Network with Global-aware Filter for Speaker Verification](https://mlsanity.com/paper/2303.11020/) (2023)
- [Improving Speaker-independent Speech Emotion Recognition Using Dynamic Joint Distribution Adaptation](https://mlsanity.com/paper/2401.09752/) (2024)
- [SISER: Speaker-Invariant Speech Emotion Recognition with Entropy-Based Adversarial Training](https://mlsanity.com/paper/2609.02941/) (2026)

## Abstract (arXiv)

Speech emotion recognition (SER) has attracted great attention in recent years due to the high demand for emotionally intelligent speech interfaces. Deriving speaker-invariant representations for speech emotion recognition is crucial. In this paper, we propose to apply adversarial training to SER to learn speaker-invariant representations. Our model consists of three parts: a representation learning sub-network with time-delay neural network (TDNN) and LSTM with statistical pooling, an emotion classification network and a speaker classification network. Both the emotion and speaker classification network take the output of the representation learning network as input. Two training strategies are employed: one based on domain adversarial training (DAT) and the other one based on cross-gradient training (CGT). Besides the conventional data set, we also evaluate our proposed models on a much larger publicly available emotion data set with 250 speakers. Evaluation results show that on IEMOCAP, DAT and CGT provides 5.6% and 7.4% improvement respectively, over a baseline system without speaker-invariant representation learning on 5-fold cross validation. On the larger emotion data set, while CGT fails to yield better results than baseline, DAT can still provide 9.8% relative improvement on a standalone test set.