# EmoNet: A Transfer Learning Framework for Multi-Corpus Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2103.08310 · HTML: https://mlsanity.com/paper/2103.08310/

## Key facts

- Cited by 17 papers in the map (data through 7 Oct 2026).
- Builds on 25 of its 122 references that are in the map.
- First posted on arXiv 10 Mar 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Efficient parametrization of multi-domain deep neural networks](https://mlsanity.com/paper/1803.10082/) (2018)
- [Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model](https://mlsanity.com/paper/1909.05330/) (2019)
- [Conversational AI: The Science Behind the Alexa Prize](https://mlsanity.com/paper/1801.03604/) (2018)
- [Attentional Pooling for Action Recognition](https://mlsanity.com/paper/1711.01467/) (2017)
- [Online Multi-Object Tracking Using CNN-based Single Object Tracker with Spatial-Temporal Attention Mechanism](https://mlsanity.com/paper/1708.02843/) (2017)
- [Attention Based Fully Convolutional Network for Speech Emotion Recognition](https://mlsanity.com/paper/1806.01506/) (2018)
- [On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognition](https://mlsanity.com/paper/2011.09212/) (2020)
- [Attentive Convolutional Neural Network based Speech Emotion Recognition: A Study on the Impact of Input Features, Signal Length, and Acted Speech](https://mlsanity.com/paper/1706.00612/) (2017)
- [Deep Architecture Enhancing Robustness to Noise, Adversarial Attacks, and Cross-corpus Setting for Speech Emotion Recognition](https://mlsanity.com/paper/2005.08453/) (2020)
- [Towards Speech Emotion Recognition "in the wild" using Aggregated Corpora and Deep Multi-Task Learning](https://mlsanity.com/paper/1708.03920/) (2017)

## Built on by (most cited first, 17 of 17)

- [Revisiting Multimodal Emotion Recognition in Conversation from the Perspective of Graph Spectrum](https://mlsanity.com/paper/2404.17862/) (2024)
- [A Comprehensive Survey on Multi-modal Conversational Emotion Recognition with Deep Learning](https://mlsanity.com/paper/2312.05735/) (2023)
- [Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization](https://mlsanity.com/paper/2304.14535/) (2023)
- [The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress](https://mlsanity.com/paper/2207.05691/) (2022)
- [COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition](https://mlsanity.com/paper/2206.05833/) (2022)
- [DeepSpectrumLite: A Power-Efficient Transfer Learning Framework for Embedded Speech and Audio Processing from Decentralised Data](https://mlsanity.com/paper/2104.11629/) (2021)
- [The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation](https://mlsanity.com/paper/2305.03369/) (2023)
- [ExHuBERT: Enhancing HuBERT Through Block Extension and Fine-Tuning on 37 Emotion Datasets](https://mlsanity.com/paper/2406.10275/) (2024)
- [Robust Federated Learning Against Adversarial Attacks for Speech Emotion Recognition](https://mlsanity.com/paper/2203.04696/) (2022)
- [Modeling Emotional Trajectories in Written Stories Utilizing Transformers and Weakly-Supervised Learning](https://mlsanity.com/paper/2406.02251/) (2024)
- [Are you sure? Analysing Uncertainty Quantification Approaches for Real-world Speech Emotion Recognition](https://mlsanity.com/paper/2407.01143/) (2024)
- [Cross-Corpora Spoken Language Identification with Domain Diversification and Generalization](https://mlsanity.com/paper/2302.05110/) (2023)
- [FaceCrafter: Identity-Conditional Diffusion with Disentangled Control over Facial Pose, Expression, and Emotion](https://mlsanity.com/paper/2505.15313/) (2025)
- [Assessing the Robustness of Spectral Clustering for Deep Speaker Diarization](https://mlsanity.com/paper/2403.14286/) (2024)
- [Fatigue Prediction in Outdoor Running Conditions using Audio Data](https://mlsanity.com/paper/2205.04343/) (2022)
- [This Paper Had the Smartest Reviewers -- Flattery Detection Utilising an Audio-Textual Transformer-Based Approach](https://mlsanity.com/paper/2406.17667/) (2024)
- [EM2LDL: A Multilingual Speech Corpus for Mixed Emotion Recognition through Label Distribution Learning](https://mlsanity.com/paper/2511.20106/) (2025)

## Abstract (arXiv)

In this manuscript, the topic of multi-corpus Speech Emotion Recognition (SER) is approached from a deep transfer learning perspective. A large corpus of emotional speech data, EmoSet, is assembled from a number of existing SER corpora. In total, EmoSet contains 84181 audio recordings from 26 SER corpora with a total duration of over 65 hours. The corpus is then utilised to create a novel framework for multi-corpus speech emotion recognition, namely EmoNet. A combination of a deep ResNet architecture and residual adapters is transferred from the field of multi-domain visual recognition to multi-corpus SER on EmoSet. Compared against two suitable baselines and more traditional training and transfer settings for the ResNet, the residual adapter approach enables parameter efficient training of a multi-domain SER model on all 26 corpora. A shared model with only 3.5 times the number of parameters of a model trained on a single database leads to increased performance for 21 of the 26 corpora in EmoSet. Measured by McNemar's test, these improvements are further significant for ten datasets at p<0.05 while there are just two corpora that see only significant decreases across the residual adapter transfer experiments. Finally, we make our EmoNet framework publicly available for users and developers at https://github.com/EIHW/EmoNet. EmoNet provides an extensive command line interface which is comprehensively documented and can be used in a variety of multi-corpus transfer learning settings.