# Multi-Task Semi-Supervised Adversarial Autoencoding for Speech Emotion Recognition

arXiv: https://arxiv.org/abs/1907.06078 · HTML: https://mlsanity.com/paper/1907.06078/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2019.

## Built on by (most cited first, 15 of 15)

- [Learning Multiple Dense Prediction Tasks from Partially Annotated Data](https://mlsanity.com/paper/2111.14893/) (2021)
- [A Comprehensive Survey on Multi-modal Conversational Emotion Recognition with Deep Learning](https://mlsanity.com/paper/2312.05735/) (2023)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Multistage linguistic conditioning of convolutional layers for speech emotion recognition](https://mlsanity.com/paper/2110.06650/) (2021)
- [Synchronizing Task Behavior: Aligning Multiple Tasks during Test-Time Training](https://mlsanity.com/paper/2507.07778/) (2025)
- [MSAC: Multiple Speech Attribute Control Method for Reliable Speech Emotion Recognition](https://mlsanity.com/paper/2308.04025/) (2023)
- [CTA-RNN: Channel and Temporal-wise Attention RNN Leveraging Pre-trained ASR Embeddings for Speech Emotion Recognition](https://mlsanity.com/paper/2203.17023/) (2022)
- [Region-aware Distribution Contrast: A Novel Approach to Multi-Task Partially Supervised Learning](https://mlsanity.com/paper/2403.10252/) (2024)
- [Representation Learning with Graph Neural Networks for Speech Emotion Recognition](https://mlsanity.com/paper/2208.09830/) (2022)
- [Speech Emotion Recognition Using Deep Sparse Auto-Encoder Extreme Learning Machine with a New Weighting Scheme and Spectro-Temporal Features Along with Classical Feature Selection and A New Quantum-Inspired Dimension Reduction Method](https://mlsanity.com/paper/2111.07094/) (2021)
- [Transfer Learning in Conversational Analysis through Reusing Preprocessing Data as Supervisors](https://mlsanity.com/paper/2112.03032/) (2021)
- [On the Efficacy and Noise-Robustness of Jointly Learned Speech Emotion and Automatic Speech Recognition](https://mlsanity.com/paper/2305.12540/) (2023)
- [Emotion-Disentangled Embedding Alignment for Noise-Robust and Cross-Corpus Speech Emotion Recognition](https://mlsanity.com/paper/2510.09072/) (2025)
- [Improving Speech Emotion Recognition with Mutual Information Regularized Generative Model](https://mlsanity.com/paper/2510.10078/) (2025)
- [Unrequited Emotions: Investigating the Gaps in Motivation and Practice in Speech Emotion Recognition Research](https://mlsanity.com/paper/2604.25776/) (2026)

## Abstract (arXiv)

Inspite the emerging importance of Speech Emotion Recognition (SER), the state-of-the-art accuracy is quite low and needs improvement to make commercial applications of SER viable. A key underlying reason for the low accuracy is the scarcity of emotion datasets, which is a challenge for developing any robust machine learning model in general. In this paper, we propose a solution to this problem: a multi-task learning framework that uses auxiliary tasks for which data is abundantly available. We show that utilisation of this additional data can improve the primary task of SER for which only limited labelled data is available. In particular, we use gender identifications and speaker recognition as auxiliary tasks, which allow the use of very large datasets, e.g., speaker classification datasets. To maximise the benefit of multi-task learning, we further use an adversarial autoencoder (AAE) within our framework, which has a strong capability to learn powerful and discriminative features. Furthermore, the unsupervised AAE in combination with the supervised classification networks enables semi-supervised learning which incorporates a discriminative component in the AAE unsupervised training pipeline. This semi-supervised learning essentially helps to improve generalisation of our framework and thus leads to improvements in SER performance. The proposed model is rigorously evaluated for categorical and dimensional emotion, and cross-corpus scenarios. Experimental results demonstrate that the proposed model achieves state-of-the-art performance on two publicly available datasets.