mlsanityData to 14 Oct 2026

NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech

arXiv 2507.13155 ↗First posted 17 Jul 2025cs.LG cs.SDcited by 14

Not on the map. Why?

Abstract (arXiv)

Current expressive speech synthesis models are constrained by the limited availability of open-source datasets containing diverse nonverbal vocalizations (NVs). In this work, we introduce NonverbalTTS (NVTTS), a 17-hour open-access dataset annotated with 10 types of NVs (e.g., laughter, coughs) and 8 emotional categories. The dataset is derived from popular sources, VoxCeleb and Expresso, using automated detection followed by human validation. We propose a comprehensive pipeline that integrates automatic speech recognition (ASR), NV tagging, emotion classification, and a fusion algorithm to merge transcriptions from multiple annotators. Fine-tuning open-source text-to-speech (TTS) models on the NVTTS dataset achieves parity with closed-source systems such as CosyVoice2, as measured by both human evaluation and automatic metrics, including speaker similarity and NV fidelity. By releasing NVTTS and its accompanying annotation guidelines, we address a key bottleneck in expressive TTS research. The dataset is available at https://huggingface.co/datasets/deepvk/NonverbalTTS.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 16 of its 32 references that are in the map. First posted on arXiv 17 Jul 2025.

Builds on

16 of 32 references.

References · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  4. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  5. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  6. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  7. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  8. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 241
  9. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
  10. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  11. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 98
  12. 2024FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applicationscited by 68
4 more
  1. 2024Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speechcited by 19
  2. 2025Scaling Rich Style-Prompted Text-to-Speech Datasetscited by 19
  3. 2024Effective Pre-Training of Audio Transformers for Sound Event Detectioncited by 12
  4. 2023JVNV: A Corpus of Japanese Emotional Speech with Verbal Content and Nonverbal Expressionscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 11.

14 papers.

Sort:

By area:

  1. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  2. 2025NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech Modeling with Paralinguistic Vocalizationscited by 15
  3. 2025MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speechcited by 6
  4. 2026NV-Bench: Benchmark of Nonverbal Vocalization Synthesis for Expressive Text-to-Speech Generationcited by 6
  5. 2026Preference Optimization for Non-Verbal Vocalization Synthesiscited by 2
  6. 2026WESR: Scaling and Evaluating Word-level Event-Speech Recognitioncited by 1
  7. 2026The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluationcited by 1
  8. 2026OpenSTBench: Beyond Semantic Evaluation for Speech Translationcited by 1
  9. 2026NVMOS: Non-Verbal Vocalization Quality Assessment in Speechcited by 1
  10. 2026Long-Tail Rebalancing for Non-Verbal Vocalization-Aware ASR: A Track 1 System for the NVVSpeech Challengecited by 1
  11. 2026MoVE: Translating Laughter and Tears via Mixture of Vocalization Experts in Speech-to-Speech Translationcited by 0
  12. 2026Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approachcited by 0
2 more
  1. 2026Modeling, Scaling, and Decoding: Optimizing Controllable Speech Generation with Nonverbal Vocalizationscited by 0
  2. 2026NVAlign: Direct-Gradient Optimization for Non-Verbal Control in Continuous Autoregressive Flow Matching Text-to-Speechcited by 0