# A Scalable Pipeline for Enabling Non-Verbal Speech Generation and Understanding

arXiv: https://arxiv.org/abs/2508.05385 · HTML: https://mlsanity.com/paper/2508.05385/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2025.

## Built on by (most cited first, 11 of 11)

- [OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models](https://mlsanity.com/paper/2604.00688/) (2026)
- [SynParaSpeech: Automated Synthesis of Paralinguistic Datasets for Speech Generation and Understanding](https://mlsanity.com/paper/2509.14946/) (2025)
- [MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speech](https://mlsanity.com/paper/2509.18196/) (2025)
- [NV-Bench: Benchmark of Nonverbal Vocalization Synthesis for Expressive Text-to-Speech Generation](https://mlsanity.com/paper/2603.15352/) (2026)
- [Preference Optimization for Non-Verbal Vocalization Synthesis](https://mlsanity.com/paper/2608.24163/) (2026)
- [WESR: Scaling and Evaluating Word-level Event-Speech Recognition](https://mlsanity.com/paper/2601.04508/) (2026)
- [NVMOS: Non-Verbal Vocalization Quality Assessment in Speech](https://mlsanity.com/paper/2606.15888/) (2026)
- [Long-Tail Rebalancing for Non-Verbal Vocalization-Aware ASR: A Track 1 System for the NVVSpeech Challenge](https://mlsanity.com/paper/2609.23462/) (2026)
- [TriA Pipeline: A Large-Scale Automatic Audio Annotation Pipeline For Audio Classification In Specific Scenarios](https://mlsanity.com/paper/2607.06179/) (2026)
- [Modeling, Scaling, and Decoding: Optimizing Controllable Speech Generation with Nonverbal Vocalizations](https://mlsanity.com/paper/2609.14231/) (2026)
- [NVAlign: Direct-Gradient Optimization for Non-Verbal Control in Continuous Autoregressive Flow Matching Text-to-Speech](https://mlsanity.com/paper/2609.31892/) (2026)

## Abstract (arXiv)

Non-verbal Vocalizations (NVs), such as laughter and sighs, are vital for conveying emotion and intention in human speech, yet most existing speech systems neglect them, which severely compromises communicative richness and emotional intelligence. Existing methods for NVs acquisition are either costly and unscalable (relying on manual annotation/recording) or unnatural (relying on rule-based synthesis). To address these limitations, we propose a highly scalable automatic annotation framework to label non-verbal phenomena from natural speech, which is low-cost, easily extendable, and inherently diverse and natural. This framework leverages a unified detection model to accurately identify NVs in natural speech and integrates them with transcripts via temporal-semantic alignment method. Using this framework, we created and released NonVerbalSpeech-38K, a diverse, real-world dataset featuring 38,718 samples across 10 NV categories collected from in-the-wild media. Experimental results demonstrate that our dataset provides superior controllability for NVs generation and achieves comparable performance for NVs understanding.