# Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings

arXiv: https://arxiv.org/abs/2104.03502 · HTML: https://mlsanity.com/paper/2104.03502/

## Key facts

- Cited by 162 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 38 references that are in the map.
- First posted on arXiv 8 Apr 2021.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition](https://mlsanity.com/paper/2008.06682/) (2020)
- [On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognition](https://mlsanity.com/paper/2011.09212/) (2020)
- [Contrastive Unsupervised Learning for Speech Emotion Recognition](https://mlsanity.com/paper/2102.06357/) (2021)
- [Transformer based unsupervised pre-training for acoustic representation learning](https://mlsanity.com/paper/2007.14602/) (2020)
- [Recognizing More Emotions with Less Data Using Self-supervised Transfer Learning](https://mlsanity.com/paper/2011.05585/) (2020)
- [General-Purpose Speech Representation Learning through a Self-Supervised Multi-Granularity Framework](https://mlsanity.com/paper/2102.01930/) (2021)

## Built on by (most cited first, 30 of 162)

- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding](https://mlsanity.com/paper/2111.02735/) (2021)
- [Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition](https://mlsanity.com/paper/2109.06870/) (2021)
- [Speech Emotion Recognition using Self-Supervised Features](https://mlsanity.com/paper/2202.03896/) (2022)
- [Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition](https://mlsanity.com/paper/2110.06309/) (2021)
- [A Large-Scale Evaluation of Speech Foundation Models](https://mlsanity.com/paper/2404.09385/) (2024)
- [FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait](https://mlsanity.com/paper/2412.01064/) (2024)
- [Fine-tuning wav2vec2 for speaker recognition](https://mlsanity.com/paper/2109.15053/) (2021)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora](https://mlsanity.com/paper/2210.02595/) (2022)
- [Detecting Dysfluencies in Stuttering Therapy Using wav2vec 2.0](https://mlsanity.com/paper/2204.03417/) (2022)
- [Multi-modal Facial Affective Analysis based on Masked Autoencoder](https://mlsanity.com/paper/2303.10849/) (2023)
- [KSoF: The Kassel State of Fluency Dataset -- A Therapy Centered Dataset of Stuttering](https://mlsanity.com/paper/2203.05383/) (2022)
- [M-SENA: An Integrated Platform for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2203.12441/) (2022)
- [Multimodal Emotion Recognition with High-level Speech and Text Features](https://mlsanity.com/paper/2111.10202/) (2021)
- [Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition](https://mlsanity.com/paper/2303.08356/) (2023)
- [Representation learning through cross-modal conditional teacher-student training for speech emotion recognition](https://mlsanity.com/paper/2112.00158/) (2021)
- [Cross-lingual Self-Supervised Speech Representations for Improved Dysarthric Speech Recognition](https://mlsanity.com/paper/2204.01670/) (2022)
- [Speaker Normalization for Self-supervised Speech Emotion Recognition](https://mlsanity.com/paper/2202.01252/) (2022)
- [Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2302.14564/) (2023)
- [Probing Speech Emotion Recognition Transformers for Linguistic Knowledge](https://mlsanity.com/paper/2204.00400/) (2022)
- [The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation](https://mlsanity.com/paper/2305.03369/) (2023)
- [Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition](https://mlsanity.com/paper/2309.10294/) (2023)
- [HCAM -- Hierarchical Cross Attention Model for Multi-modal Emotion Recognition](https://mlsanity.com/paper/2304.06910/) (2023)
- [ExHuBERT: Enhancing HuBERT Through Block Extension and Fine-Tuning on 37 Emotion Datasets](https://mlsanity.com/paper/2406.10275/) (2024)
- [CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech](https://mlsanity.com/paper/2506.02863/) (2025)
- [Fully Automated End-to-End Fake Audio Detection](https://mlsanity.com/paper/2208.09618/) (2022)
- [GEmo-CLAP: Gender-Attribute-Enhanced Contrastive Language-Audio Pretraining for Accurate Speech Emotion Recognition](https://mlsanity.com/paper/2306.07848/) (2023)

## Abstract (arXiv)

Emotion recognition datasets are relatively small, making the use of the more sophisticated deep learning approaches challenging. In this work, we propose a transfer learning method for speech emotion recognition where features extracted from pre-trained wav2vec 2.0 models are modeled using simple neural networks. We propose to combine the output of several layers from the pre-trained model using trainable weights which are learned jointly with the downstream model. Further, we compare performance using two different wav2vec 2.0 models, with and without finetuning for speech recognition. We evaluate our proposed approaches on two standard emotion databases IEMOCAP and RAVDESS, showing superior performance compared to results in the literature.