# Towards Generalizable SER: Soft Labeling and Data Augmentation for Modeling Temporal Emotion Shifts in Large-Scale Multilingual Speech

arXiv: https://arxiv.org/abs/2311.08607 · HTML: https://mlsanity.com/paper/2311.08607/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 27 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [Multi-attention Recurrent Network for Human Communication Comprehension](https://mlsanity.com/paper/1802.00923/) (2018)
- [The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systems](https://mlsanity.com/paper/1806.09514/) (2018)
- [Cross Lingual Speech Emotion Recognition: Urdu vs. Western Languages](https://mlsanity.com/paper/1812.10411/) (2018)
- [STUDIES: Corpus of Japanese Empathetic Dialogue Speech Towards Friendly Voice Agent](https://mlsanity.com/paper/2203.14757/) (2022)
- [The ACM Multimedia 2023 Computational Paralinguistics Challenge: Emotion Share & Requests](https://mlsanity.com/paper/2304.14882/) (2023)
- [Pre-Finetuning for Few-Shot Emotional Speech Recognition](https://mlsanity.com/paper/2302.12921/) (2023)

## Built on by (most cited first, 2 of 2)

- [SER Evals: In-domain and Out-of-domain Benchmarking for Speech Emotion Recognition](https://mlsanity.com/paper/2408.07851/) (2024)
- [Learning More with Less: Self-Supervised Approaches for Low-Resource Speech Emotion Recognition](https://mlsanity.com/paper/2506.02059/) (2025)

## Abstract (arXiv)

Recognizing emotions in spoken communication is crucial for advanced human-machine interaction. Current emotion detection methodologies often display biases when applied cross-corpus. To address this, our study amalgamates 16 diverse datasets, resulting in 375 hours of data across languages like English, Chinese, and Japanese. We propose a soft labeling system to capture gradational emotional intensities. Using the Whisper encoder and data augmentation methods inspired by contrastive learning, our method emphasizes the temporal dynamics of emotions. Our validation on four multilingual datasets demonstrates notable zero-shot generalization. We publish our open source model weights and initial promising results after fine-tuning on Hume-Prosody.