# EmoNet-Voice: A Fine-Grained, Expert-Verified Benchmark for Speech Emotion Detection

arXiv: https://arxiv.org/abs/2506.09827 · HTML: https://mlsanity.com/paper/2506.09827/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 39 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Builds on (most cited first, 15 of 15)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research](https://mlsanity.com/paper/2005.00357/) (2020)
- [LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Models](https://mlsanity.com/paper/2406.05113/) (2024)
- [Why human-AI relationships need socioaffective alignment](https://mlsanity.com/paper/2502.02528/) (2025)
- [EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmark](https://mlsanity.com/paper/2406.07162/) (2024)
- [The MuSe 2021 Multimodal Sentiment Analysis Challenge: Sentiment, Emotion, Physiological-Emotion, and Stress](https://mlsanity.com/paper/2104.07123/) (2021)
- [SERAB: A multi-lingual benchmark for speech emotion recognition](https://mlsanity.com/paper/2110.03414/) (2021)
- [Does CLIP Know My Face?](https://mlsanity.com/paper/2209.07341/) (2022)
- [ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognition](https://mlsanity.com/paper/2305.16065/) (2023)
- [SER Evals: In-domain and Out-of-domain Benchmarking for Speech Emotion Recognition](https://mlsanity.com/paper/2408.07851/) (2024)
- [EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition](https://mlsanity.com/paper/2505.20033/) (2025)
- [BERSting at the Screams: A Benchmark for Distanced, Emotional and Shouted Speech Recognition](https://mlsanity.com/paper/2505.00059/) (2025)

## Built on by (most cited first, 3 of 3)

- [Bridging What the Model Thinks and How It Speaks: Expressive Speech Generation via Self-Aware Intent-Realization Alignment](https://mlsanity.com/paper/2604.11424/) (2026)
- [SpeechSense: A Paralinguistic-Focused Dataset for Fine-Grained Speech Sentiment Analysis](https://mlsanity.com/paper/2608.17931/) (2026)
- [VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale](https://mlsanity.com/paper/2609.32016/) (2026)

## Abstract (arXiv)

Speech emotion recognition (SER) systems are constrained by existing datasets that typically cover only 6-10 basic emotions, lack scale and diversity, and face ethical challenges when collecting sensitive emotional states. We introduce EMONET-VOICE, a comprehensive resource addressing these limitations through two components: (1) EmoNet-Voice Big, a 5,000-hour multilingual pre-training dataset spanning 40 fine-grained emotion categories across 11 voices and 4 languages, and (2) EmoNet-Voice Bench, a rigorously validated benchmark of 4,7k samples with unanimous expert consensus on emotion presence and intensity levels. Using state-of-the-art synthetic voice generation, our privacy-preserving approach enables ethical inclusion of sensitive emotions (e.g., pain, shame) while maintaining controlled experimental conditions. Each sample underwent validation by three psychology experts. We demonstrate that our Empathic Insight models trained on our synthetic data achieve strong real-world dataset generalization, as tested on EmoDB and RAVDESS. Furthermore, our comprehensive evaluation reveals that while high-arousal emotions (e.g., anger: 95% accuracy) are readily detected, the benchmark successfully exposes the difficulty of distinguishing perceptually similar emotions (e.g., sadness vs. distress: 63% discrimination), providing quantifiable metrics for advancing nuanced emotion AI. EMONET-VOICE establishes a new paradigm for large-scale, ethically-sourced, fine-grained SER research.