# Best Practices for Noise-Based Augmentation to Improve the Performance of Deployable Speech-Based Emotion Recognition Systems

arXiv: https://arxiv.org/abs/2104.08806 · HTML: https://mlsanity.com/paper/2104.08806/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 43 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 14 of 14)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Audio Adversarial Examples: Targeted Attacks on Speech-to-Text](https://mlsanity.com/paper/1801.01944/) (2018)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [Improving the Robustness of Deep Neural Networks via Stability Training](https://mlsanity.com/paper/1604.04326/) (2016)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Practical Hidden Voice Attacks against Speech and Speaker Recognition Systems](https://mlsanity.com/paper/1904.05734/) (2019)
- [A Hybrid DSP/Deep Learning Approach to Real-Time Full-Band Speech Enhancement](https://mlsanity.com/paper/1709.08243/) (2017)
- [CopyPaste: An Augmentation Method for Speech Emotion Recognition](https://mlsanity.com/paper/2010.14602/) (2020)
- [MuSE-ing on the Impact of Utterance Ordering On Crowdsourced Emotion Annotations](https://mlsanity.com/paper/1903.11672/) (2019)
- [Capturing Long-term Temporal Dependencies with Convolutional Networks for Continuous Emotion Recognition](https://mlsanity.com/paper/1708.07050/) (2017)
- [Siamese Capsule Network for End-to-End Speaker Recognition In The Wild](https://mlsanity.com/paper/2009.13480/) (2020)
- [A Study of All-Convolutional Encoders for Connectionist Temporal Classification](https://mlsanity.com/paper/1710.10398/) (2017)
- [Replay spoofing detection system for automatic speaker verification using multi-task learning of noise classes](https://mlsanity.com/paper/1808.09638/) (2018)

## Built on by (most cited first, 1 of 1)

- [Implicit Design Choices and Their Impact on Emotion Recognition Model Development and Evaluation](https://mlsanity.com/paper/2309.03238/) (2023)

## Abstract (arXiv)

Speech emotion recognition is an important component of any human centered system. But speech characteristics produced and perceived by a person can be influenced by a multitude of reasons, both desirable such as emotion, and undesirable such as noise. To train robust emotion recognition models, we need a large, yet realistic data distribution, but emotion datasets are often small and hence are augmented with noise. Often noise augmentation makes one important assumption, that the prediction label should remain the same in presence or absence of noise, which is true for automatic speech recognition but not necessarily true for perception based tasks. In this paper we make three novel contributions. We validate through crowdsourcing that the presence of noise does change the annotation label and hence may alter the original ground truth label. We then show how disregarding this knowledge and assuming consistency in ground truth labels propagates to downstream evaluation of ML models, both for performance evaluation and robustness testing. We end the paper with a set of recommendations for noise augmentations in speech emotion recognition datasets.