# On the Efficacy and Noise-Robustness of Jointly Learned Speech Emotion and Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2305.12540 · HTML: https://mlsanity.com/paper/2305.12540/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 21 May 2023.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [Fusing ASR Outputs in Joint Training for Speech Emotion Recognition](https://mlsanity.com/paper/2110.15684/) (2021)
- [Interactive Feature Fusion for End-to-End Noise-Robust Speech Recognition](https://mlsanity.com/paper/2110.05267/) (2021)
- [Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition](https://mlsanity.com/paper/2201.06309/) (2022)
- [Multi-Task Semi-Supervised Adversarial Autoencoding for Speech Emotion Recognition](https://mlsanity.com/paper/1907.06078/) (2019)
- [Multitask-Based Joint Learning Approach To Robust ASR For Radio Communication Speech](https://mlsanity.com/paper/2107.10701/) (2021)
- [On the Robustness of Speech Emotion Recognition for Human-Robot Interaction with Deep Neural Networks](https://mlsanity.com/paper/1804.02173/) (2018)
- [Towards Speech Emotion Recognition "in the wild" using Aggregated Corpora and Deep Multi-Task Learning](https://mlsanity.com/paper/1708.03920/) (2017)
- [Adversarial Joint Training with Self-Attention Mechanism for Robust End-to-End Speech Recognition](https://mlsanity.com/paper/2104.01471/) (2021)

## Abstract (arXiv)

New-age conversational agent systems perform both speech emotion recognition (SER) and automatic speech recognition (ASR) using two separate and often independent approaches for real-world application in noisy environments. In this paper, we investigate a joint ASR-SER multitask learning approach in a low-resource setting and show that improvements are observed not only in SER, but also in ASR. We also investigate the robustness of such jointly trained models to the presence of background noise, babble, and music. Experimental results on the IEMOCAP dataset show that joint learning can improve ASR word error rate (WER) and SER classification accuracy by 10.7% and 2.3% respectively in clean scenarios. In noisy scenarios, results on data augmented with MUSAN show that the joint approach outperforms the independent ASR and SER approaches across many noisy conditions. Overall, the joint ASR-SER approach yielded more noise-resistant models than the independent ASR and SER approaches.