mlsanityData to 7 Oct 2026

A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding

arXiv 2111.02735 ↗First posted 4 Nov 2021cs.CL cs.NE cs.SDcited by 82

Not on the map. Why?

Abstract (arXiv)

Speech self-supervised models such as wav2vec 2.0 and HuBERT are making revolutionary progress in Automatic Speech Recognition (ASR). However, they have not been totally proven to produce better performance on tasks other than ASR. In this work, we explored partial fine-tuning and entire fine-tuning on wav2vec 2.0 and HuBERT pre-trained models for three non-ASR speech tasks: Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding. With simple proposed downstream frameworks, the best scores reached 79.58% weighted accuracy on speaker-dependent setting and 73.01% weighted accuracy on speaker-independent setting for Speech Emotion Recognition on IEMOCAP, 2.36% equal error rate for Speaker Verification on VoxCeleb1, 89.38% accuracy for Intent Classification and 78.92% F1 for Slot Filling on SLURP, showing the strength of fine-tuned wav2vec 2.0 and HuBERT on learning prosodic, voice-print and semantic representations.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 4 Nov 2021.

Builds on

27 of 39 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  6. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  8. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  9. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  10. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  11. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  12. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
15 more
  1. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  2. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  3. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  4. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  5. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 158
  6. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  7. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 93
  8. 2018Towards end-to-end spoken language understandingcited by 92
  9. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 76
  10. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  11. 2021Fine-tuning wav2vec2 for speaker recognitioncited by 35
  12. 2021Integration of Pre-trained Networks with Continuous Token Interface for End-to-End Spoken Language Understandingcited by 22
  13. 2019Recent Advances in End-to-End Spoken Language Understandingcited by 13
  14. 2021Timers and Such: A Practical Benchmark for Spoken Language Understanding with Numberscited by 11
  15. 2020Siamese Capsule Network for End-to-End Speaker Recognition In The Wildcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 31; 2024: 9; 2025: 19; 2026: 8.

82 papers.

Sort:

By area:

  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  2. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  3. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 155
  4. 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 79
  5. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 62
  6. 2023Efficient Sequence Transduction by Jointly Predicting Tokens and Durationscited by 56
  7. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
  8. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
  9. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  10. 2023Can ChatGPT Detect Intent? Evaluating Large Language Models for Spoken Language Understandingcited by 21
  11. 2023AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Modelcited by 16
  12. 2023Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Headscited by 16
38 more
  1. 2022Probing Speech Emotion Recognition Transformers for Linguistic Knowledgecited by 13
  2. 2023Speech Emotion Diarization: Which Emotion Appears When?cited by 13
  3. 2023Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognitioncited by 12
  4. 2022Phoneme Segmentation Using Self-Supervised Speech Modelscited by 11
  5. 2024ExHuBERT: Enhancing HuBERT Through Block Extension and Fine-Tuning on 37 Emotion Datasetscited by 11
  6. 2022Disentangling Prosody Representations with Unsupervised Speech Reconstructioncited by 8
  7. 2023A vector quantized masked autoencoder for speech emotion recognitioncited by 8
  8. 2024SLIM: Style-Linguistics Mismatch Model for Generalized Audio Deepfake Detectioncited by 8
  9. 2022Hierarchical Softmax for End-to-End Low-resource Multilingual Speech Recognitioncited by 7
  10. 2022MTI-Net: A Multi-Target Speech Intelligibility Prediction Modelcited by 6
  11. 2023Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAPcited by 6
  12. 2023A vector quantized masked autoencoder for audiovisual speech emotion recognitioncited by 6
  13. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillationcited by 6
  14. 2024A Layer-Anchoring Strategy for Enhancing Cross-Lingual Speech Emotion Recognitioncited by 6
  15. 2024Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of Adapterscited by 5
  16. 2025BERSting at the Screams: A Benchmark for Distanced, Emotional and Shouted Speech Recognitioncited by 5
  17. 2025Deep Learning Approaches for Multimodal Intent Recognition: A Surveycited by 5
  18. 2025EDTalk++: Full Disentanglement for Controllable Talking Head Synthesiscited by 5
  19. 2022Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillationcited by 4
  20. 2023The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translationcited by 4
  21. 2023Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognitioncited by 4
  22. 2023Speech Emotion Recognition with Distilled Prosodic and Linguistic Affect Representationscited by 4
  23. 2025Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Informationcited by 4
  24. 2025EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognitioncited by 4
  25. 2022Skit-S2I: An Indian Accented Speech to Intent datasetcited by 3
  26. 2023Self-supervised Neural Factor Analysis for Disentangling Utterance-level Speech Representationscited by 3
  27. 2023CIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Trainingcited by 3
  28. 2023Learning Multilingual Expressive Speech Representation for Prosody Prediction without Parallel Datacited by 3
  29. 2023Knowledge-Aware Audio-Grounded Generative Slot Filling for Limited Annotated Datacited by 3
  30. 2023Leveraging Pretrained ASR Encoders for Effective and Efficient End-to-End Speech Intent Classification and Slot Fillingcited by 3
  31. 2023Leveraging Label Information for Multimodal Emotion Recognitioncited by 3
  32. 2023Hierarchical Audio-Visual Information Fusion with Multi-label Joint Decoding for MER 2023cited by 3
  33. 2024MSP-Podcast SER Challenge 2024: L'antenne du Ventoux Multimodal Self-Supervised Learning for Speech Emotion Recognitioncited by 3
  34. 2025Efficient Finetuning for Dimensional Speech Emotion Recognition in the Age of Transformerscited by 3
  35. 2025Can Diffusion Models Disentangle? A Theoretical Perspectivecited by 3
  36. 2023Enhancing expressivity transfer in textless speech-to-speech translationcited by 2
  37. 2025Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalizationcited by 2
  38. 2025Accent-Invariant Automatic Speech Recognition via Saliency-Driven Spectrogram Maskingcited by 2

50 most cited of 82 All 82 (JSON)