# Attention-Augmented End-to-End Multi-Task Learning for Emotion Prediction from Speech

arXiv: https://arxiv.org/abs/1903.12424 · HTML: https://mlsanity.com/paper/1903.12424/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Mar 2019.

## Built on by (most cited first, 17 of 17)

- [FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio](https://mlsanity.com/paper/2403.01901/) (2024)
- [Audio-Visual Deception Detection: DOLOS Dataset and Parameter-Efficient Crossmodal Learning](https://mlsanity.com/paper/2303.12745/) (2023)
- [Cross-Modal Perceptionist: Can Face Geometry be Gleaned from Voices?](https://mlsanity.com/paper/2203.09824/) (2022)
- [An Improved StarGAN for Emotional Voice Conversion: Enhancing Voice Quality and Data Augmentation](https://mlsanity.com/paper/2107.08361/) (2021)
- [MSAC: Multiple Speech Attribute Control Method for Reliable Speech Emotion Recognition](https://mlsanity.com/paper/2308.04025/) (2023)
- [FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles](https://mlsanity.com/paper/2501.03181/) (2025)
- [Voice2Mesh: Cross-Modal 3D Face Model Generation from Voices](https://mlsanity.com/paper/2104.10299/) (2021)
- [STG-MTL: Scalable Task Grouping for Multi-Task Learning Using Data Map](https://mlsanity.com/paper/2307.03374/) (2023)
- [Rethinking Voice-Face Correlation: A Geometry View](https://mlsanity.com/paper/2307.13948/) (2023)
- [Towards Principled Task Grouping for Multi-Task Learning](https://mlsanity.com/paper/2402.15328/) (2024)
- [Dynamic Restrained Uncertainty Weighting Loss for Multitask Learning of Vocal Expression](https://mlsanity.com/paper/2206.11049/) (2022)
- [Evaluating Deep Music Generation Methods Using Data Augmentation](https://mlsanity.com/paper/2201.00052/) (2021)
- [Propagating Variational Model Uncertainty for Bioacoustic Call Label Smoothing](https://mlsanity.com/paper/2210.10526/) (2022)
- [STAA-Net: A Sparse and Transferable Adversarial Attack for Speech Emotion Recognition](https://mlsanity.com/paper/2402.01227/) (2024)
- [The Hidden Dance of Phonemes and Visage: Unveiling the Enigmatic Link between Phonemes and Facial Features](https://mlsanity.com/paper/2307.13953/) (2023)
- [Emotion Embeddings unicodex2014 Learning Stable and Homogeneous Abstractions from Heterogeneous Affective Datasets](https://mlsanity.com/paper/2308.07871/) (2023)
- [Task Selection and Assignment for Multi-modal Multi-task Dialogue Act Classification with Non-stationary Multi-armed Bandits](https://mlsanity.com/paper/2309.09832/) (2023)

## Abstract (arXiv)

Despite the increasing research interest in end-to-end learning systems for speech emotion recognition, conventional systems either suffer from the overfitting due in part to the limited training data, or do not explicitly consider the different contributions of automatically learnt representations for a specific task. In this contribution, we propose a novel end-to-end framework which is enhanced by learning other auxiliary tasks and an attention mechanism. That is, we jointly train an end-to-end network with several different but related emotion prediction tasks, i.e., arousal, valence, and dominance predictions, to extract more robust representations shared among various tasks than traditional systems with the hope that it is able to relieve the overfitting problem. Meanwhile, an attention layer is implemented on top of the layers for each task, with the aim to capture the contribution distribution of different segment parts for each individual task. To evaluate the effectiveness of the proposed system, we conducted a set of experiments on the widely used database IEMOCAP. The empirical results show that the proposed systems significantly outperform corresponding baseline systems.