mlsanityData to 14 Oct 2026

Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings

arXiv 2104.03502 ↗First posted 8 Apr 2021cs.SD cs.LG eess.AScited by 162

Not on the map. Why?

Abstract (arXiv)

Emotion recognition datasets are relatively small, making the use of the more sophisticated deep learning approaches challenging. In this work, we propose a transfer learning method for speech emotion recognition where features extracted from pre-trained wav2vec 2.0 models are modeled using simple neural networks. We propose to combine the output of several layers from the pre-trained model using trainable weights which are learned jointly with the downstream model. Further, we compare performance using two different wav2vec 2.0 models, with and without finetuning for speech recognition. We evaluate our proposed approaches on two standard emotion databases IEMOCAP and RAVDESS, showing superior performance compared to results in the literature.

Key facts. Cited by 162 papers in the map (data through 14 Oct 2026). Builds on 13 of its 38 references that are in the map. First posted on arXiv 8 Apr 2021.

Builds on

13 of 38 references.

References · 13

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  5. 2018Deep contextualized word representationscited by 2,290
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  7. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  8. 2020Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognitioncited by 38
  9. 2020On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognitioncited by 19
  10. 2021Contrastive Unsupervised Learning for Speech Emotion Recognitioncited by 19
  11. 2020Transformer based unsupervised pre-training for acoustic representation learningcited by 10
  12. 2020Recognizing More Emotions with Less Data Using Self-supervised Transfer Learningcited by 7
1 more
  1. 2021General-Purpose Speech Representation Learning through a Self-Supervised Multi-Granularity Frameworkcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 39; 2023: 37; 2024: 30; 2025: 28; 2026: 19.

162 papers.

Sort:

By area:

  1. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
  2. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 157
  3. 2021Unsupervised Speech Recognitioncited by 148
  4. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  5. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  6. 2022Speech Emotion Recognition using Self-Supervised Featurescited by 53
  7. 2021Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognitioncited by 49
  8. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 40
  9. 2024FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portraitcited by 40
  10. 2021Fine-tuning wav2vec2 for speaker recognitioncited by 39
  11. 2023Transformers in Speech Processing: A Surveycited by 39
  12. 2022Exploration of A Self-Supervised Speech Model: A Study on Emotional Corporacited by 31
38 more
  1. 2022Detecting Dysfluencies in Stuttering Therapy Using wav2vec 2.0cited by 27
  2. 2023Multi-modal Facial Affective Analysis based on Masked Autoencodercited by 23
  3. 2022KSoF: The Kassel State of Fluency Dataset -- A Therapy Centered Dataset of Stutteringcited by 21
  4. 2022M-SENA: An Integrated Platform for Multimodal Sentiment Analysiscited by 21
  5. 2021Multimodal Emotion Recognition with High-level Speech and Text Featurescited by 20
  6. 2023Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognitioncited by 20
  7. 2021Representation learning through cross-modal conditional teacher-student training for speech emotion recognitioncited by 17
  8. 2022Cross-lingual Self-Supervised Speech Representations for Improved Dysarthric Speech Recognitioncited by 17
  9. 2022Speaker Normalization for Self-supervised Speech Emotion Recognitioncited by 14
  10. 2023Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognitioncited by 14
  11. 2022Probing Speech Emotion Recognition Transformers for Linguistic Knowledgecited by 13
  12. 2023The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisationcited by 12
  13. 2023Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognitioncited by 12
  14. 2023HCAM -- Hierarchical Cross Attention Model for Multi-modal Emotion Recognitioncited by 11
  15. 2024ExHuBERT: Enhancing HuBERT Through Block Extension and Fine-Tuning on 37 Emotion Datasetscited by 11
  16. 2025CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speechcited by 11
  17. 2022Fully Automated End-to-End Fake Audio Detectioncited by 10
  18. 2023GEmo-CLAP: Gender-Attribute-Enhanced Contrastive Language-Audio Pretraining for Accurate Speech Emotion Recognitioncited by 10
  19. 2021Multi-task Voice Activated Framework using Self-supervised Learningcited by 9
  20. 2022The Influence of Dataset Partitioning on Dysfluency Detection Systemscited by 9
  21. 2022Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognitioncited by 9
  22. 2023Using Auxiliary Tasks In Multimodal Fusion Of Wav2vec 2.0 And BERT For Multimodal Emotion Recognitioncited by 9
  23. 2024Self-supervised ASR Models and Features For Dysarthric and Elderly Speech Recognitioncited by 9
  24. 2022Anti-Spoofing Using Transfer Learning with Variational Information Bottleneckcited by 8
  25. 2023Knowledge-aware Bayesian Co-attention for Multimodal Emotion Recognitioncited by 8
  26. 2023Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognitioncited by 8
  27. 2023A vector quantized masked autoencoder for speech emotion recognitioncited by 8
  28. 2024Revealing Emotional Clusters in Speaker Embeddings: A Contrastive Learning Strategy for Speech Emotion Recognitioncited by 8
  29. 2024SLIM: Style-Linguistics Mismatch Model for Generalized Audio Deepfake Detectioncited by 8
  30. 2022Learning Speech Emotion Representations in the Quaternion Domaincited by 7
  31. 2022On the Use of Semantically-Aligned Speech Representations for Spoken Language Understandingcited by 7
  32. 2022Fast Yet Effective Speech Emotion Recognition with Self-distillationcited by 7
  33. 2023A vector quantized masked autoencoder for audiovisual speech emotion recognitioncited by 7
  34. 2024Automated Assessment of Encouragement and Warmth in Classrooms Leveraging Multimodal Emotional Features and ChatGPTcited by 7
  35. 2022Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillationcited by 6
  36. 2023Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAPcited by 6
  37. 2023A multimodal dynamical variational autoencoder for audiovisual speech representation learningcited by 6
  38. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillationcited by 6

50 most cited of 162 All 162 (JSON)