# EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmark

arXiv: https://arxiv.org/abs/2406.07162 · HTML: https://mlsanity.com/paper/2406.07162/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 20 of 20)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset](https://mlsanity.com/paper/2010.14794/) (2020)
- [Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language](https://mlsanity.com/paper/2212.07525/) (2022)
- [The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systems](https://mlsanity.com/paper/1806.09514/) (2018)
- [M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database](https://mlsanity.com/paper/2205.10237/) (2022)
- [Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations](https://mlsanity.com/paper/2402.12786/) (2024)
- [MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning](https://mlsanity.com/paper/2304.08981/) (2023)
- [Cross Lingual Speech Emotion Recognition: Urdu vs. Western Languages](https://mlsanity.com/paper/1812.10411/) (2018)
- [SERAB: A multi-lingual benchmark for speech emotion recognition](https://mlsanity.com/paper/2110.03414/) (2021)
- [ShEMO -- A Large-Scale Validated Database for Persian Speech Emotion Detection](https://mlsanity.com/paper/1906.01155/) (2019)
- [Climate and Weather: Inspecting Depression Detection via Emotion Recognition](https://mlsanity.com/paper/2204.14099/) (2022)
- [EMNS /Imz/ Corpus: An emotive single-speaker dataset for narrative storytelling in games, television and graphic novels](https://mlsanity.com/paper/2305.13137/) (2023)
- [Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAP](https://mlsanity.com/paper/2304.00860/) (2023)
- [A New Amharic Speech Emotion Dataset and Classification Benchmark](https://mlsanity.com/paper/2201.02710/) (2022)

## Built on by (most cited first, 30 of 40)

- [CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training](https://mlsanity.com/paper/2505.17589/) (2025)
- [FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs](https://mlsanity.com/paper/2407.04051/) (2024)
- [Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models](https://mlsanity.com/paper/2503.02318/) (2025)
- [MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation](https://mlsanity.com/paper/2412.04448/) (2024)
- [Differentiable Reward Optimization for LLM based TTS system](https://mlsanity.com/paper/2507.05911/) (2025)
- [Improving Speech Emotion Recognition in Under-Resourced Languages via Speech-to-Speech Translation with Bootstrapping Data Selection](https://mlsanity.com/paper/2409.10985/) (2024)
- [MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speech](https://mlsanity.com/paper/2509.18196/) (2025)
- [Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thought](https://mlsanity.com/paper/2502.18186/) (2025)
- [Contextual Paralinguistic Data Creation for Multi-Modal Speech-LLM: Data Condensation and Spoken QA Generation](https://mlsanity.com/paper/2505.13338/) (2025)
- [TouchASP: Elastic Automatic Speech Perception that Everyone Can Touch](https://mlsanity.com/paper/2412.15622/) (2024)
- [Can Diffusion Models Disentangle? A Theoretical Perspective](https://mlsanity.com/paper/2504.00220/) (2025)
- [EmoNet-Voice: A Fine-Grained, Expert-Verified Benchmark for Speech Emotion Detection](https://mlsanity.com/paper/2506.09827/) (2025)
- [Marco-Voice Technical Report](https://mlsanity.com/paper/2508.02038/) (2025)
- [VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs](https://mlsanity.com/paper/2603.08936/) (2026)
- [Something from Nothing: Data Augmentation for Robust Severity Level Estimation of Dysarthric Speech](https://mlsanity.com/paper/2603.15988/) (2026)
- [CAMEO: Collection of Multilingual Emotional Speech Corpora](https://mlsanity.com/paper/2505.11051/) (2025)
- [VoxAging: Continuously Tracking Speaker Aging with a Large-Scale Longitudinal Dataset in English and Mandarin](https://mlsanity.com/paper/2505.21445/) (2025)
- [EmoCAST: Emotional Talking Portrait via Emotive Text Description](https://mlsanity.com/paper/2508.20615/) (2025)
- [RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesis](https://mlsanity.com/paper/2510.14628/) (2025)
- [Speech Emotion Recognition Leveraging OpenAI's Whisper Representations and Attentive Pooling Methods](https://mlsanity.com/paper/2602.06000/) (2026)
- [Multilingual Emotion Neurons in Large Audio-Language Models](https://mlsanity.com/paper/2608.08772/) (2026)
- [Represented but Ignored: A Causal Account of Prosodic Underuse in Audio-Language Models](https://mlsanity.com/paper/2608.19211/) (2026)
- [Leveraging Cross-Attention Transformer and Multi-Feature Fusion for Cross-Linguistic Speech Emotion Recognition](https://mlsanity.com/paper/2501.10408/) (2025)
- [Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languages](https://mlsanity.com/paper/2503.21806/) (2025)
- [Seeing What You Say: Expressive Image Generation from Speech](https://mlsanity.com/paper/2511.03423/) (2025)
- [MERaLiON-SER: Robust Speech Emotion Recognition Model for English and SEA Languages](https://mlsanity.com/paper/2511.04914/) (2025)
- [Foundation Model-based Evaluation of Neuropsychiatric Disorders: A Lifespan-Inclusive, Multi-Modal, and Multi-Lingual Study](https://mlsanity.com/paper/2512.20948/) (2025)
- [Attention-weighted Centered Kernel Alignment for Knowledge Distillation in Large Audio-Language Models Applied to Speech Emotion Recognition](https://mlsanity.com/paper/2602.01547/) (2026)
- [Semantics-Aware Generative Latent Data Augmentation for Learning in Low-Resource Domains](https://mlsanity.com/paper/2602.02841/) (2026)
- [Color-based Emotion Representation for Speech Emotion Recognition](https://mlsanity.com/paper/2602.16256/) (2026)

## Abstract (arXiv)

Speech emotion recognition (SER) is an important part of human-computer interaction, receiving extensive attention from both industry and academia. However, the current research field of SER has long suffered from the following problems: 1) There are few reasonable and universal splits of the datasets, making comparing different models and methods difficult. 2) No commonly used benchmark covers numerous corpus and languages for researchers to refer to, making reproduction a burden. In this paper, we propose EmoBox, an out-of-the-box multilingual multi-corpus speech emotion recognition toolkit, along with a benchmark for both intra-corpus and cross-corpus settings. For intra-corpus settings, we carefully designed the data partitioning for different datasets. For cross-corpus settings, we employ a foundation SER model, emotion2vec, to mitigate annotation errors and obtain a test set that is fully balanced in speakers and emotions distributions. Based on EmoBox, we present the intra-corpus SER results of 10 pre-trained speech models on 32 emotion datasets with 14 languages, and the cross-corpus SER results on 4 datasets with the fully balanced test sets. To the best of our knowledge, this is the largest SER benchmark, across language scopes and quantity scales. We hope that our toolkit and benchmark can facilitate the research of SER in the community.