mlsanityData to 14 Oct 2026

Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognition

arXiv 2307.10757 ↗First posted 20 Jul 2023cs.SD cs.CL eess.AScited by 16

Not on the map. Why?

Abstract (arXiv)

This paper presents a paradigm that adapts general large-scale pretrained models (PTMs) to speech emotion recognition task. Although PTMs shed new light on artificial general intelligence, they are constructed with general tasks in mind, and thus, their efficacy for specific tasks can be further improved. Additionally, employing PTMs in practical applications can be challenging due to their considerable size. Above limitations spawn another research direction, namely, optimizing large-scale PTMs for specific tasks to generate task-specific PTMs that are both compact and effective. In this paper, we focus on the speech emotion recognition task and propose an improved emotion-specific pretrained encoder called Vesper. Vesper is pretrained on a speech dataset based on WavLM and takes into account emotional characteristics. To enhance sensitivity to emotional information, Vesper employs an emotion-guided masking strategy to identify the regions that need masking. Subsequently, Vesper employs hierarchical and cross-layer self-supervision to improve its ability to capture acoustic and semantic representations, both of which are crucial for emotion recognition. Experimental results on the IEMOCAP, MELD, and CREMA-D datasets demonstrate that Vesper with 4 layers outperforms WavLM Base with 12 layers, and the performance of Vesper with 12 layers surpasses that of WavLM Large with 24 layers.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 53 of its 86 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

53 of 86 references.

References · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  10. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  11. 2023Segment Anythingcited by 8,234
  12. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
38 more
  1. 2016Layer Normalizationcited by 5,669
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  10. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  11. 2020Image Segmentation Using Deep Learning: A Surveycited by 660
  12. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  13. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  14. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  15. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  16. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
  17. 2020Toward Transformer-Based Object Detectioncited by 84
  18. 2020RSPNet: Relative Speed Perception for Unsupervised Video Representation Learningcited by 83
  19. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  20. 2022Audio Self-supervised Learning: A Surveycited by 65
  21. 2020Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentationcited by 53
  22. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  23. 2020AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Searchcited by 51
  24. 2021Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognitioncited by 49
  25. 2022Masked Motion Encoding for Self-Supervised Video Representation Learningcited by 39
  26. 2021Deep Learning-based Non-Intrusive Multi-Objective Speech Assessment Model with Cross-Domain Featurescited by 33
  27. 2022Speech Emotion Recognition with Co-Attention based Multi-level Acoustic Informationcited by 32
  28. 2019Acquiring Knowledge from Pre-trained Model to Neural Machine Translationcited by 30
  29. 2020DeFormer: Decomposing Pre-trained Transformers for Faster Question Answeringcited by 29
  30. 2021Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoderscited by 24
  31. 2022SpeechFormer: A Hierarchical Efficient Framework Incorporating the Characteristics of Speechcited by 20
  32. 2020BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognitioncited by 19
  33. 2021LSSED: a large-scale dataset and benchmark for speech emotion recognitioncited by 17
  34. 2023SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processingcited by 16
  35. 2021Speech Emotion Recognition using Semantic Informationcited by 11
  36. 2023DST: Deformable Speech Transformer for Emotion Recognitioncited by 11
  37. 2021Key-Sparse Transformer for Multimodal Speech Emotion Recognitioncited by 10
  38. 2021CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Modelscited by 5

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 6; 2026: 3.

16 papers.

Sort:

By area:

  1. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
  2. 2023Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognitioncited by 12
  3. 2025MAVEN: Multi-modal Attention for Valence-Arousal Emotion Networkcited by 8
  4. 2024Adapting WavLM for Speech Emotion Recognitioncited by 4
  5. 2023Active Learning Based Fine-Tuning Framework for Speech Emotion Recognitioncited by 2
  6. 2024Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attentioncited by 2
  7. 2025Multi-Teacher Language-Aware Knowledge Distillation for Multilingual Speech Emotion Recognitioncited by 2
  8. 2025Bridging Minds and Machines: Toward an Integration of AI and Cognitive Sciencecited by 2
  9. 2024Active Learning with Task Adaptation Pre-training for Speech Emotion Recognitioncited by 0
  10. 2024Searching for Effective Preprocessing Method and CNN-based Architecture with Efficient Channel Attention on Speech Emotion Recognitioncited by 0
  11. 2025Leveraging Cross-Attention Transformer and Multi-Feature Fusion for Cross-Linguistic Speech Emotion Recognitioncited by 0
  12. 2025Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languagescited by 0
4 more
  1. 2025EmoSphere-SER: Enhancing Speech Emotion Recognition Through Spherical Representation with Auxiliary Classificationcited by 0
  2. 2026Color-based Emotion Representation for Speech Emotion Recognitioncited by 0
  3. 2026Explainable Lightweight Compact Deep Models for Speech Emotion Recognitioncited by 0
  4. 2026Dual-Scale State-Space Modeling with Speaker-Wise Dynamic CRF for Speech Emotion Recognition in Conversationcited by 0