# Cross-Lingual Speech Emotion Recognition: Humans vs. Self-Supervised Models

arXiv: https://arxiv.org/abs/2409.16920 · HTML: https://mlsanity.com/paper/2409.16920/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Builds on (most cited first, 18 of 18)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora](https://mlsanity.com/paper/2210.02595/) (2022)
- [Cross Lingual Speech Emotion Recognition: Urdu vs. Western Languages](https://mlsanity.com/paper/1812.10411/) (2018)
- [Fusing ASR Outputs in Joint Training for Speech Emotion Recognition](https://mlsanity.com/paper/2110.15684/) (2021)
- [PEFT-SER: On the Use of Parameter Efficient Transfer Learning Approaches For Speech Emotion Recognition Using Pre-trained Speech Models](https://mlsanity.com/paper/2306.05350/) (2023)
- [Speech Emotion Diarization: Which Emotion Appears When?](https://mlsanity.com/paper/2306.12991/) (2023)
- [Cross-lingual and Multilingual Speech Emotion Recognition on English and French](https://mlsanity.com/paper/1803.00357/) (2018)
- [Unsupervised Adversarial Domain Adaptation for Cross-Lingual Speech Emotion Recognition](https://mlsanity.com/paper/1907.06083/) (2019)
- [Layer-Wise Analysis of Self-Supervised Acoustic Word Embeddings: A Study on Speech Emotion Recognition](https://mlsanity.com/paper/2402.02617/) (2024)
- [A Layer-Anchoring Strategy for Enhancing Cross-Lingual Speech Emotion Recognition](https://mlsanity.com/paper/2407.04966/) (2024)
- [Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognition](https://mlsanity.com/paper/2308.08713/) (2023)
- [Unsupervised Cross-Lingual Speech Emotion Recognition Using DomainAdversarial Neural Network](https://mlsanity.com/paper/2012.11174/) (2020)
- [Parameter Efficient Finetuning for Speech Emotion Recognition and Domain Adaptation](https://mlsanity.com/paper/2402.11747/) (2024)

## Built on by (most cited first, 3 of 3)

- [Multilingual Emotion Neurons in Large Audio-Language Models](https://mlsanity.com/paper/2608.08772/) (2026)
- [Exploring Acoustic Similarity in Emotional Speech and Music via Self-Supervised Representations](https://mlsanity.com/paper/2409.17899/) (2024)
- [Disentangling Acoustic Cues in Alzheimer's Pathology and Perception: The Roles of Language and Gender](https://mlsanity.com/paper/2607.23977/) (2026)

## Abstract (arXiv)

Utilizing Self-Supervised Learning (SSL) models for Speech Emotion Recognition (SER) has proven effective, yet limited research has explored cross-lingual scenarios. This study presents a comparative analysis between human performance and SSL models, beginning with a layer-wise analysis and an exploration of parameter-efficient fine-tuning strategies in monolingual, cross-lingual, and transfer learning contexts. We further compare the SER ability of models and humans at both utterance- and segment-levels. Additionally, we investigate the impact of dialect on cross-lingual SER through human evaluation. Our findings reveal that models, with appropriate knowledge transfer, can adapt to the target language and achieve performance comparable to native speakers. We also demonstrate the significant effect of dialect on SER for individuals without prior linguistic and paralinguistic background. Moreover, both humans and models exhibit distinct behaviors across different emotions. These results offer new insights into the cross-lingual SER capabilities of SSL models, underscoring both their similarities to and differences from human emotion perception.