# Improved Speech Emotion Recognition using Transfer Learning and Spectrogram Augmentation

arXiv: https://arxiv.org/abs/2108.02510 · HTML: https://mlsanity.com/paper/2108.02510/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 51 references that are in the map.
- First posted on arXiv 5 Aug 2021.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Multimodal Speech Emotion Recognition Using Audio and Text](https://mlsanity.com/paper/1810.04635/) (2018)
- [BUT System Description to VoxCeleb Speaker Recognition Challenge 2019](https://mlsanity.com/paper/1910.12592/) (2019)
- [Emotion Recognition in Speech using Cross-Modal Transfer in the Wild](https://mlsanity.com/paper/1808.05561/) (2018)
- [x-vectors meet emotions: A study on dependencies between emotion and speaker recognition](https://mlsanity.com/paper/2002.05039/) (2020)
- [Emotion recognition by fusing time synchronous and time asynchronous representations](https://mlsanity.com/paper/2010.14102/) (2020)
- [Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning](https://mlsanity.com/paper/1804.05788/) (2018)
- [Improving Cross-Corpus Speech Emotion Recognition with Adversarial Discriminative Domain Generalization (ADDoG)](https://mlsanity.com/paper/1903.12094/) (2019)
- [Attentive Convolutional Neural Network based Speech Emotion Recognition: A Study on the Impact of Input Features, Signal Length, and Acted Speech](https://mlsanity.com/paper/1706.00612/) (2017)
- [Convolutional RNN: an Enhanced Model for Extracting Features from Sequential Data](https://mlsanity.com/paper/1602.05875/) (2016)
- [CNN+LSTM Architecture for Speech Emotion Recognition with Data Augmentation](https://mlsanity.com/paper/1802.05630/) (2018)
- [Progressive Neural Networks for Transfer Learning in Emotion Recognition](https://mlsanity.com/paper/1706.03256/) (2017)
- [Transfer Learning for Improving Speech Emotion Classification Accuracy](https://mlsanity.com/paper/1801.06353/) (2018)
- [Emotion Recognition From Speech With Recurrent Neural Networks](https://mlsanity.com/paper/1701.08071/) (2017)

## Built on by (most cited first, 12 of 12)

- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization](https://mlsanity.com/paper/2304.14535/) (2023)
- [Non-Contrastive Self-supervised Learning for Utterance-Level Information Extraction from Speech](https://mlsanity.com/paper/2208.05445/) (2022)
- [Revealing Emotional Clusters in Speaker Embeddings: A Contrastive Learning Strategy for Speech Emotion Recognition](https://mlsanity.com/paper/2401.11017/) (2024)
- [Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models](https://mlsanity.com/paper/2202.08974/) (2022)
- [Privacy-preserving Speech Emotion Recognition through Semi-Supervised Federated Learning](https://mlsanity.com/paper/2202.02611/) (2022)
- [Active Learning Based Fine-Tuning Framework for Speech Emotion Recognition](https://mlsanity.com/paper/2310.00283/) (2023)
- [LineConGraphs: Line Conversation Graphs for Effective Emotion Recognition using Graph Neural Networks](https://mlsanity.com/paper/2312.03756/) (2023)
- [NeuroFlux: Memory-Efficient CNN Training Using Adaptive Local Learning](https://mlsanity.com/paper/2402.14139/) (2024)
- [Active Learning with Task Adaptation Pre-training for Speech Emotion Recognition](https://mlsanity.com/paper/2405.00307/) (2024)
- [Improving Robustness of Spectrogram Classifiers with Neural Stochastic Differential Equations](https://mlsanity.com/paper/2409.01532/) (2024)

## Abstract (arXiv)

Automatic speech emotion recognition (SER) is a challenging task that plays a crucial role in natural human-computer interaction. One of the main challenges in SER is data scarcity, i.e., insufficient amounts of carefully labeled data to build and fully explore complex deep learning models for emotion classification. This paper aims to address this challenge using a transfer learning strategy combined with spectrogram augmentation. Specifically, we propose a transfer learning approach that leverages a pre-trained residual network (ResNet) model including a statistics pooling layer from speaker recognition trained using large amounts of speaker-labeled data. The statistics pooling layer enables the model to efficiently process variable-length input, thereby eliminating the need for sequence truncation which is commonly used in SER systems. In addition, we adopt a spectrogram augmentation technique to generate additional training data samples by applying random time-frequency masks to log-mel spectrograms to mitigate overfitting and improve the generalization of emotion recognition models. We evaluate the effectiveness of our proposed approach on the interactive emotional dyadic motion capture (IEMOCAP) dataset. Experimental results indicate that the transfer learning and spectrogram augmentation approaches improve the SER performance, and when combined achieve state-of-the-art results.