# Speaker-invariant Affective Representation Learning via Adversarial Training

arXiv: https://arxiv.org/abs/1911.01533 · HTML: https://mlsanity.com/paper/1911.01533/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2019.

## Built on by (most cited first, 14 of 14)

- [Multimodal Emotion Recognition with High-level Speech and Text Features](https://mlsanity.com/paper/2111.10202/) (2021)
- [Disentangling Voice and Content with Self-Supervision for Speaker Recognition](https://mlsanity.com/paper/2310.01128/) (2023)
- [You Are What You Write: Preserving Privacy in the Era of Large Language Models](https://mlsanity.com/paper/2204.09391/) (2022)
- [MMER: Multimodal Multi-task Learning for Speech Emotion Recognition](https://mlsanity.com/paper/2203.16794/) (2022)
- [Robust Stuttering Detection via Multi-task and Adversarial Learning](https://mlsanity.com/paper/2204.01735/) (2022)
- [Introducing ECAPA-TDNN and Wav2Vec2.0 Embeddings to Stuttering Detection](https://mlsanity.com/paper/2204.01564/) (2022)
- [Stuttering Detection Using Speaker Representations and Self-supervised Contextual Embeddings](https://mlsanity.com/paper/2306.00689/) (2023)
- [Exploring speaker enrolment for few-shot personalisation in emotional vocalisation prediction](https://mlsanity.com/paper/2206.06680/) (2022)
- [Acted vs. Improvised: Domain Adaptation for Elicitation Approaches in Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2104.01978/) (2021)
- [Unsupervised Speech Representation Learning for Behavior Modeling using Triplet Enhanced Contextualized Networks](https://mlsanity.com/paper/2104.03899/) (2021)
- [DSNet: Disentangled Siamese Network with Neutral Calibration for Speech Emotion Recognition](https://mlsanity.com/paper/2312.15593/) (2023)
- [Speech Emotion Recognition Using Deep Sparse Auto-Encoder Extreme Learning Machine with a New Weighting Scheme and Spectro-Temporal Features Along with Classical Feature Selection and A New Quantum-Inspired Dimension Reduction Method](https://mlsanity.com/paper/2111.07094/) (2021)
- [Machine Unlearning for Speaker-Agnostic Detection of Gender-Based Violence Condition in Speech](https://mlsanity.com/paper/2411.18177/) (2024)
- [SISER: Speaker-Invariant Speech Emotion Recognition with Entropy-Based Adversarial Training](https://mlsanity.com/paper/2609.02941/) (2026)

## Abstract (arXiv)

Representation learning for speech emotion recognition is challenging due to labeled data sparsity issue and lack of gold standard references. In addition, there is much variability from input speech signals, human subjective perception of the signals and emotion label ambiguity. In this paper, we propose a machine learning framework to obtain speech emotion representations by limiting the effect of speaker variability in the speech signals. Specifically, we propose to disentangle the speaker characteristics from emotion through an adversarial training network in order to better represent emotion. Our method combines the gradient reversal technique with an entropy loss function to remove such speaker information. Our approach is evaluated on both IEMOCAP and CMU-MOSEI datasets. We show that our method improves speech emotion classification and increases generalization to unseen speakers.