# Multitask Learning from Augmented Auxiliary Data for Improving Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2207.05298 · HTML: https://mlsanity.com/paper/2207.05298/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jul 2022.

## Built on by (most cited first, 11 of 11)

- [DialogueLLM: Context and Emotion Knowledge-Tuned Large Language Models for Emotion Recognition in Conversations](https://mlsanity.com/paper/2310.11374/) (2023)
- [Natural Language Processing in Electronic Health Records in Relation to Healthcare Decision-making: A Systematic Review](https://mlsanity.com/paper/2306.12834/) (2023)
- [Refashioning Emotion Recognition Modelling: The Advent of Generalised Large Models](https://mlsanity.com/paper/2308.11578/) (2023)
- [CFN-ESA: A Cross-Modal Fusion Network with Emotion-Shift Awareness for Dialogue Emotion Recognition](https://mlsanity.com/paper/2307.15432/) (2023)
- [Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition](https://mlsanity.com/paper/2309.10294/) (2023)
- [A Preliminary Study on Augmenting Speech Emotion Recognition using a Diffusion Model](https://mlsanity.com/paper/2305.11413/) (2023)
- [Enhancing Speech Emotion Recognition Through Differentiable Architecture Search](https://mlsanity.com/paper/2305.14402/) (2023)
- [Cross-Language Speech Emotion Recognition Using Multimodal Dual Attention Transformers](https://mlsanity.com/paper/2306.13804/) (2023)
- [LineConGraphs: Line Conversation Graphs for Effective Emotion Recognition using Graph Neural Networks](https://mlsanity.com/paper/2312.03756/) (2023)
- [Integrating Contrastive Learning into a Multitask Transformer Model for Effective Domain Adaptation](https://mlsanity.com/paper/2310.04703/) (2023)
- [emoDARTS: Joint Optimisation of CNN & Sequential Neural Network Architectures for Superior Speech Emotion Recognition](https://mlsanity.com/paper/2403.14083/) (2024)

## Abstract (arXiv)

Despite the recent progress in speech emotion recognition (SER), state-of-the-art systems lack generalisation across different conditions. A key underlying reason for poor generalisation is the scarcity of emotion datasets, which is a significant roadblock to designing robust machine learning (ML) models. Recent works in SER focus on utilising multitask learning (MTL) methods to improve generalisation by learning shared representations. However, most of these studies propose MTL solutions with the requirement of meta labels for auxiliary tasks, which limits the training of SER systems. This paper proposes an MTL framework (MTL-AUG) that learns generalised representations from augmented data. We utilise augmentation-type classification and unsupervised reconstruction as auxiliary tasks, which allow training SER systems on augmented data without requiring any meta labels for auxiliary tasks. The semi-supervised nature of MTL-AUG allows for the exploitation of the abundant unlabelled data to further boost the performance of SER. We comprehensively evaluate the proposed framework in the following settings: (1) within corpus, (2) cross-corpus and cross-language, (3) noisy speech, (4) and adversarial attacks. Our evaluations using the widely used IEMOCAP, MSP-IMPROV, and EMODB datasets show improved results compared to existing state-of-the-art methods.