mlsanityData to 14 Oct 2026

Speech Emotion Recognition using Self-Supervised Features

arXiv 2202.03896 ↗First posted 7 Feb 2022cs.SD cs.AI cs.LGcited by 53

Not on the map. Why?

Abstract (arXiv)

Self-supervised pre-trained features have consistently delivered state-of-art results in the field of natural language processing (NLP); however, their merits in the field of speech emotion recognition (SER) still need further investigation. In this paper we introduce a modular End-to- End (E2E) SER system based on an Upstream + Downstream architecture paradigm, which allows easy use/integration of a large variety of self-supervised features. Several SER experiments for predicting categorical emotion classes from the IEMOCAP dataset are performed. These experiments investigate interactions among fine-tuning of self-supervised feature models, aggregation of frame-level features into utterance-level features and back-end classification networks. The proposed monomodal speechonly based system not only achieves SOTA results, but also brings light to the possibility of powerful and well finetuned self-supervised acoustic features that reach results similar to the results achieved by SOTA multimodal systems using both Speech and Text modalities.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 7 Feb 2022.

Builds on

19 of 26 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  5. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  6. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  7. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  8. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  9. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 162
  10. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  11. 2018Training Tips for the Transformer Modelcited by 85
  12. 2019M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cuescited by 56
7 more
  1. 2020Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognitioncited by 38
  2. 2020Emotion recognition by fusing time synchronous and time asynchronous representationscited by 26
  3. 2018Multimodal Sentiment Analysis: Addressing Key Issues and Setting up the Baselinescited by 23
  4. 2019Speech Emotion Recognition with Dual-Sequence LSTM Architecturecited by 23
  5. 2019Speech Sentiment Analysis via Pre-trained Features from End-to-end ASR Modelscited by 21
  6. 2019Speech Emotion Recognition Using Multi-hop Attention Mechanismcited by 18
  7. 2018Transfer Learning for Improving Speech Emotion Classification Accuracycited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 14; 2024: 17; 2025: 10; 2026: 6.

53 papers.

Sort:

By area:

  1. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
  2. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  3. 2024Temporal-Channel Modeling in Multi-head Self-Attention for Synthetic Speech Detectioncited by 25
  4. 2023Layer Collaboration in the Forward-Forward Algorithmcited by 21
  5. 2023Self-supervised representations in speech-based depression detectioncited by 18
  6. 2024On the social bias of speech self-supervised modelscited by 16
  7. 2023The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisationcited by 12
  8. 2023Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognitioncited by 12
  9. 2023GEmo-CLAP: Gender-Attribute-Enhanced Contrastive Language-Audio Pretraining for Accurate Speech Emotion Recognitioncited by 10
  10. 2023Using Auxiliary Tasks In Multimodal Fusion Of Wav2vec 2.0 And BERT For Multimodal Emotion Recognitioncited by 9
  11. 2024Revealing Emotional Clusters in Speaker Embeddings: A Contrastive Learning Strategy for Speech Emotion Recognitioncited by 8
  12. 2024Layer-Wise Analysis of Self-Supervised Acoustic Word Embeddings: A Study on Speech Emotion Recognitioncited by 8
38 more
  1. 2022MMER: Multimodal Multi-task Learning for Speech Emotion Recognitioncited by 7
  2. 2024Efficient Feature Extraction and Late Fusion Strategy for Audiovisual Emotional Mimicry Intensity Estimationcited by 6
  3. 2023Learning Robust Self-attention Features for Speech Emotion Recognition with Label-adaptive Mixupcited by 5
  4. 2024Interface Design for Self-Supervised Speech Modelscited by 5
  5. 2022Knowledge Transfer For On-Device Speech Emotion Recognition with Neural Structured Learningcited by 4
  6. 2023Effect of Attention and Self-Supervised Speech Embeddings on Non-Semantic Speech Taskscited by 4
  7. 2023Transferring speech-generic and depression-specific knowledge for Alzheimer's disease detectioncited by 4
  8. 2024Adapting WavLM for Speech Emotion Recognitioncited by 4
  9. 2024Spontaneous Speech-Based Suicide Risk Detection Using Whisper and Large Language Modelscited by 4
  10. 2023Exploring Emotion Expression Recognition in Older Adults Interacting with a Virtual Coachcited by 3
  11. 2024Parameter Efficient Finetuning for Speech Emotion Recognition and Domain Adaptationcited by 3
  12. 2024The MuSe 2024 Multimodal Sentiment Analysis Challenge: Social Perception and Humor Recognitioncited by 3
  13. 2022Self-supervision and Learnable STRFs for Age, Emotion, and Country Predictioncited by 2
  14. 2022Distribution-based Emotion Recognition in Conversationcited by 2
  15. 2023Active Learning Based Fine-Tuning Framework for Speech Emotion Recognitioncited by 2
  16. 2022Unifying the Discrete and Continuous Emotion labels for Speech Emotion Recognitioncited by 1
  17. 2023Cross-Language Speech Emotion Recognition Using Multimodal Dual Attention Transformerscited by 1
  18. 2023Evaluating raw waveforms with deep learning frameworks for speech emotion recognitioncited by 1
  19. 2024STAA-Net: A Sparse and Transferable Adversarial Attack for Speech Emotion Recognitioncited by 1
  20. 2024Are Paralinguistic Representations all that is needed for Speech Emotion Recognition?cited by 1
  21. 2024Investigating Acoustic-Textual Emotional Inconsistency Information for Automatic Depression Detectioncited by 1
  22. 2025Foundation Model Hidden Representations for Heart Rate Estimation from Auscultationcited by 1
  23. 2025Layer-Wise Analysis of Self-Supervised Representations for Age and Gender Classification in Children's Speechcited by 1
  24. 2025SSL-SE-EEG: A Framework for Robust Learning from Unlabeled EEG Data with Self-Supervised Learning and Squeeze-Excitation Networkscited by 1
  25. 2022Application of Knowledge Distillation to Multi-task Speech Representation Learningcited by 0
  26. 2024Active Learning with Task Adaptation Pre-training for Speech Emotion Recognitioncited by 0
  27. 2024Exploring Self-Supervised Multi-view Contrastive Learning for Speech Emotion Recognition with Limited Annotationscited by 0
  28. 2024Emotion Loss Attacking: Adversarial Attack Perception for Skeleton based on Multi-dimensional Featurescited by 0
  29. 2024Can We Estimate Purchase Intention Based on Zero-shot Speech Emotion Recognition?cited by 0
  30. 2025Residual Speech Embeddings for Tone Classification: Removing Linguistic Content to Enhance Paralinguistic Analysiscited by 0
  31. 2025PARROT: Synergizing Mamba and Attention-based SSL Pre-Trained Models via Parallel Branch Hadamard Optimal Transport for Speech Emotion Recognitioncited by 0
  32. 2025Enhancing Speech Emotion Recognition with Graph-Based Multimodal Fusion and Prosodic Features for the Speech Emotion Recognition in Naturalistic Conditions Challenge at Interspeech 2025cited by 0
  33. 2025Leveraging Unlabeled Audio-Visual Data in Speech Emotion Recognition using Knowledge Distillationcited by 0
  34. 2025Transformer Redesign for Late Fusion of Audio-Text Features on Ultra-Low-Power Edge Hardwarecited by 0
  35. 2025Unifying Model and Layer Fusion for Speech Foundation Modelscited by 0
  36. 2026Multimodal Sentiment Analysis based on Multi-channel and Symmetric Mutual Promotion Feature Fusioncited by 0
  37. 2026EMO-BOOST: Emotion-Augmented Audio-Visual Features for Improved Generalization in Deepfake Detectioncited by 0
  38. 2026How Well Do Self-Supervised Speech Models Encode Age and Gender in Children's Speech? A Layer-Wise Analysis Across Multiple Architecturescited by 0

50 most cited of 53 All 53 (JSON)