# NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech

arXiv: https://arxiv.org/abs/2507.13155 · HTML: https://mlsanity.com/paper/2507.13155/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 17 Jul 2025.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens](https://mlsanity.com/paper/2407.05407/) (2024)
- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation](https://mlsanity.com/paper/2407.05361/) (2024)
- [EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis](https://mlsanity.com/paper/2308.05725/) (2023)
- [FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications](https://mlsanity.com/paper/2409.03283/) (2024)
- [Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2407.12229/) (2024)
- [Scaling Rich Style-Prompted Text-to-Speech Datasets](https://mlsanity.com/paper/2503.04713/) (2025)
- [Effective Pre-Training of Audio Transformers for Sound Event Detection](https://mlsanity.com/paper/2409.09546/) (2024)
- [JVNV: A Corpus of Japanese Emotional Speech with Verbal Content and Nonverbal Expressions](https://mlsanity.com/paper/2310.06072/) (2023)

## Built on by (most cited first, 14 of 14)

- [InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue](https://mlsanity.com/paper/2510.13747/) (2025)
- [NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech Modeling with Paralinguistic Vocalizations](https://mlsanity.com/paper/2508.04195/) (2025)
- [MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speech](https://mlsanity.com/paper/2509.18196/) (2025)
- [NV-Bench: Benchmark of Nonverbal Vocalization Synthesis for Expressive Text-to-Speech Generation](https://mlsanity.com/paper/2603.15352/) (2026)
- [Preference Optimization for Non-Verbal Vocalization Synthesis](https://mlsanity.com/paper/2608.24163/) (2026)
- [WESR: Scaling and Evaluating Word-level Event-Speech Recognition](https://mlsanity.com/paper/2601.04508/) (2026)
- [The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation](https://mlsanity.com/paper/2604.26347/) (2026)
- [OpenSTBench: Beyond Semantic Evaluation for Speech Translation](https://mlsanity.com/paper/2605.30792/) (2026)
- [NVMOS: Non-Verbal Vocalization Quality Assessment in Speech](https://mlsanity.com/paper/2606.15888/) (2026)
- [Long-Tail Rebalancing for Non-Verbal Vocalization-Aware ASR: A Track 1 System for the NVVSpeech Challenge](https://mlsanity.com/paper/2609.23462/) (2026)
- [MoVE: Translating Laughter and Tears via Mixture of Vocalization Experts in Speech-to-Speech Translation](https://mlsanity.com/paper/2604.17435/) (2026)
- [Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach](https://mlsanity.com/paper/2606.21215/) (2026)
- [Modeling, Scaling, and Decoding: Optimizing Controllable Speech Generation with Nonverbal Vocalizations](https://mlsanity.com/paper/2609.14231/) (2026)
- [NVAlign: Direct-Gradient Optimization for Non-Verbal Control in Continuous Autoregressive Flow Matching Text-to-Speech](https://mlsanity.com/paper/2609.31892/) (2026)

## Abstract (arXiv)

Current expressive speech synthesis models are constrained by the limited availability of open-source datasets containing diverse nonverbal vocalizations (NVs). In this work, we introduce NonverbalTTS (NVTTS), a 17-hour open-access dataset annotated with 10 types of NVs (e.g., laughter, coughs) and 8 emotional categories. The dataset is derived from popular sources, VoxCeleb and Expresso, using automated detection followed by human validation. We propose a comprehensive pipeline that integrates automatic speech recognition (ASR), NV tagging, emotion classification, and a fusion algorithm to merge transcriptions from multiple annotators. Fine-tuning open-source text-to-speech (TTS) models on the NVTTS dataset achieves parity with closed-source systems such as CosyVoice2, as measured by both human evaluation and automatic metrics, including speaker similarity and NV fidelity. By releasing NVTTS and its accompanying annotation guidelines, we address a key bottleneck in expressive TTS research. The dataset is available at https://huggingface.co/datasets/deepvk/NonverbalTTS.