mlsanityData to 14 Oct 2026

Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognition

arXiv 2308.08713 ↗First posted 17 Aug 2023cs.CL cs.SD eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Recent advancements in transformer-based speech representation models have greatly transformed speech processing. However, there has been limited research conducted on evaluating these models for speech emotion recognition (SER) across multiple languages and examining their internal representations. This article addresses these gaps by presenting a comprehensive benchmark for SER with eight speech representation models and six different languages. We conducted probing experiments to gain insights into inner workings of these models for SER. We find that using features from a single optimal layer of a speech model reduces the error rate by 32% on average across seven datasets when compared to systems where features from all layers of speech models are used. We also achieve state-of-the-art results for German and Persian languages. Our probing results indicate that the middle layers of speech models capture the most important emotional information for speech emotion recognition.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 17 Aug 2023.

Builds on

24 of 33 references.

References · 24

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  5. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  6. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  7. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  8. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 162
  9. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  10. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  11. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 71
  12. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
12 more
  1. 2021Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognitioncited by 49
  2. 2022On the Utility of Self-supervised Models for Prosody-related Taskscited by 42
  3. 2022Exploration of A Self-Supervised Speech Model: A Study on Emotional Corporacited by 31
  4. 2022Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?cited by 27
  5. 2021SERAB: A multi-lingual benchmark for speech emotion recognitioncited by 22
  6. 2019ShEMO -- A Large-Scale Validated Database for Persian Speech Emotion Detectioncited by 21
  7. 2020Acoustics Based Intent Recognition Using Discovered Phonetic Units for Low Resource Languagescited by 8
  8. 2023A vector quantized masked autoencoder for speech emotion recognitioncited by 8
  9. 2022Emotion Recognition In Persian Speech Using Deep Neural Networkscited by 7
  10. 2021Intent Recognition and Unsupervised Slot Identification for Low Resourced Spoken Dialog Systemscited by 4
  11. 2022On Building Spoken Language Understanding Systems for Low Resourced Languagescited by 4
  12. 2022Multilingual Speech Emotion Recognition With Multi-Gating Mechanism and Neural Architecture Searchcited by 1

Built on by

Cited by year

Citing papers per year: 2024: 4.

4 papers.

  1. 2024Improving Semantic Understanding in Speech Language Models via Brain-tuningcited by 20
  2. 2024Cross-Lingual Speech Emotion Recognition: Humans vs. Self-Supervised Modelscited by 3
  3. 2024Explaining Deep Learning Embeddings for Speech Emotion Recognition by Predicting Interpretable Acoustic Featurescited by 2
  4. 2024Are Paralinguistic Representations all that is needed for Speech Emotion Recognition?cited by 1