mlsanityData to 7 Oct 2026

Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition

arXiv 2110.06309 ↗First posted 12 Oct 2021eess.AS cs.CL cs.LGcited by 47

Not on the map. Why?

Abstract (arXiv)

While Wav2Vec 2.0 has been proposed for speech recognition (ASR), it can also be used for speech emotion recognition (SER); its performance can be significantly improved using different fine-tuning strategies. Two baseline methods, vanilla fine-tuning (V-FT) and task adaptive pretraining (TAPT) are first presented. We show that V-FT is able to outperform state-of-the-art models on the IEMOCAP dataset. TAPT, an existing NLP fine-tuning strategy, further improves the performance on SER. We also introduce a novel fine-tuning method termed P-TAPT, which modifies the TAPT objective to learn contextualized emotion representations. Experiments show that P-TAPT performs better than TAPT, especially under low-resource settings. Compared to prior works in this literature, our top-line system achieved a 7.4% absolute improvement in unweighted accuracy (UA) over the state-of-the-art performance on IEMOCAP. Our code is publicly available.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 15 of its 30 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

15 of 30 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  9. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 158
  10. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  11. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 93
  12. 2018Attention Based Fully Convolutional Network for Speech Emotion Recognitioncited by 23
3 more
  1. 2018CNN+LSTM Architecture for Speech Emotion Recognition with Data Augmentationcited by 17
  2. 2020Recognizing More Emotions with Less Data Using Self-supervised Transfer Learningcited by 7
  3. 2020Advancing Multiple Instance Learning with Attention Modeling for Categorical Speech Emotion Recognitioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 13; 2024: 16; 2025: 10; 2026: 2.

47 papers.

Sort:

By area:

  1. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 155
  2. 2023FedMultimodal: A Benchmark For Multimodal Federated Learningcited by 51
  3. 2022Exploration of A Self-Supervised Speech Model: A Study on Emotional Corporacited by 31
  4. 2024HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognitioncited by 30
  5. 2024Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applicationscited by 25
  6. 2022M-SENA: An Integrated Platform for Multimodal Sentiment Analysiscited by 21
  7. 2024Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modalitycited by 21
  8. 2023Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognitioncited by 16
  9. 2022A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairnesscited by 13
  10. 2023Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognitioncited by 12
  11. 2022Improving Mispronunciation Detection with Wav2vec2-based Momentum Pseudo-Labeling for Accentedness and Intelligibility Assessmentcited by 10
  12. 2023GEmo-CLAP: Gender-Attribute-Enhanced Contrastive Language-Audio Pretraining for Accurate Speech Emotion Recognitioncited by 10
35 more
  1. 2023A vector quantized masked autoencoder for speech emotion recognitioncited by 8
  2. 2023Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detectioncited by 7
  3. 2023A vector quantized masked autoencoder for audiovisual speech emotion recognitioncited by 6
  4. 2024Unimodal Multi-Task Fusion for Emotional Mimicry Intensity Predictioncited by 6
  5. 2024A Layer-Anchoring Strategy for Enhancing Cross-Lingual Speech Emotion Recognitioncited by 6
  6. 2025Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thoughtcited by 5
  7. 2023Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognitioncited by 4
  8. 2025Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Informationcited by 4
  9. 2023Leveraging Label Information for Multimodal Emotion Recognitioncited by 3
  10. 2025Efficient Finetuning for Dimensional Speech Emotion Recognition in the Age of Transformerscited by 3
  11. 2023Active Learning Based Fine-Tuning Framework for Speech Emotion Recognitioncited by 2
  12. 2023Frame-level emotional state alignment method for speech emotion recognitioncited by 2
  13. 2024GMP-TL: Gender-augmented Multi-scale Pseudo-label Enhanced Transfer Learning for Speech Emotion Recognitioncited by 2
  14. 2024Two-stage Framework for Robust Speech Emotion Recognition Using Target Speaker Extraction in Human Speech Noise Conditionscited by 2
  15. 2025Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalizationcited by 2
  16. 2025Multi-Teacher Language-Aware Knowledge Distillation for Multilingual Speech Emotion Recognitioncited by 2
  17. 2025Pushing the Performance of Synthetic Speech Detection with Kolmogorov-Arnold Networks and Self-Supervised Learning Modelscited by 2
  18. 2022An Efficient Multitask Learning Architecture for Affective Vocal Burst Analysiscited by 1
  19. 2023PhasePerturbation: Speech Data Augmentation via Phase Perturbation for Automatic Speech Recognitioncited by 1
  20. 2024End to end Hindi to English speech conversion using Bark, mBART and a finetuned XLSR Wav2Vec2cited by 1
  21. 2024STAA-Net: A Sparse and Transferable Adversarial Attack for Speech Emotion Recognitioncited by 1
  22. 2024What Does it Take to Generalize SER Model Across Datasets? A Comprehensive Benchmarkcited by 1
  23. 2024Investigating Acoustic-Textual Emotional Inconsistency Information for Automatic Depression Detectioncited by 1
  24. 2025FedAPM: Federated Learning via ADMM with Partial Model Personalizationcited by 1
  25. 2023A Comparison of Time-based Models for Multimodal Emotion Recognitioncited by 0
  26. 2024Transcription and translation of videos using fine-tuned XLSR Wav2Vec2 on custom dataset and mBARTcited by 0
  27. 2024Active Learning with Task Adaptation Pre-training for Speech Emotion Recognitioncited by 0
  28. 2024Exploring Self-Supervised Multi-view Contrastive Learning for Speech Emotion Recognition with Limited Annotationscited by 0
  29. 2024Can We Estimate Purchase Intention Based on Zero-shot Speech Emotion Recognition?cited by 0
  30. 2024NoLoR: An ASR-Based Framework for Expedited Endangered Language Documentation with Neo-Aramaic as a Case Studycited by 0
  31. 2025Leveraging Cross-Attention Transformer and Multi-Feature Fusion for Cross-Linguistic Speech Emotion Recognitioncited by 0
  32. 2025HarmoniFuse: A Component-Selective and Prompt-Adaptive Framework for Multi-Task Speech Language Modelingcited by 0
  33. 2025Multi-Loss Learning for Speech Emotion Recognition with Energy-Adaptive Mixup and Frame-Level Attentioncited by 0
  34. 2026Geometric Second-Order Feature Correlation Learning for Self-Supervised Speech Emotion Recognitioncited by 0
  35. 2026Enriching Speech Emotion Representations with Conversational Contextcited by 0