Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue
On the map: Speech and audio language models → Emotional speech synthesis and recognition also fits: Audio language models and spoken dialogue
Abstract (arXiv)
The user experience of speech interaction models has improved, benefiting from the modeling of paralinguistic attributes and advances in speech reconstruction paradigms. However, human communication follows extensive norms regarding non-verbal vocalizations (NVVs) spanning respiratory, affective, and discourse. The omission of these events imposes inherent limitations on the naturalness of current approaches. In this work, we cover 13 major types of NVVs and construct a natural dialogue dataset that conforms to human conversational norms through an automated synthesis pipeline. We enhance the ability of speech interaction models to generate contextually appropriate NVV events through multi-stage fine-tuning and NVV-aware GRPO (Group Relative Policy Optimization). Based on distinct modeling paradigms, we substantially improve the expressiveness of VocalNet and VITA-Qinyu in empathy interactions and role-play dialogues, and further discuss the architectural necessity for achieving high-quality natural speech generation.
Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Emotional speech synthesis and recognition, one of 11 directions in Speech and audio language models. Builds on 13 of its 18 references that are in the map. First posted on arXiv 19 Oct 2026.
Builds on
13 of 18 references.
Closest ancestors · 2
Same direction, newest first
- 2025MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speech19 Sep
- 2025NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech17 Jul
Other references · 11
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,110
- 2025Qwen3-Omni Technical Reportcited by 578
- 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 552
- 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 215
- 2026Qwen3-ASR Technical Reportcited by 162
- 2020A Large-Scale Chinese Short-Text Conversation Datasetcited by 78
- 2025MiniMax-Speech: Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encodercited by 60
- 2025A Scalable Pipeline for Enabling Non-Verbal Speech Generation and Understandingcited by 16
- 2024DisfluencySpeech -- Single-Speaker Conversational Speech Dataset with Paralanguagecited by 9
- 2026JoyAI-Talker: Full-Duplex Speech Interactive Large Model Built for Empathetic Voice Agentscited by 5
- 2026VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singingcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026NV-Bench: Benchmark of Nonverbal Vocalization Synthesis for Expressive Text-to-Speech Generation
- 2025NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech Modeling with Paralinguistic Vocalizations
- 2026Toward Natural Emotional Text-To-Speech System with Fine-Grained Non-Verbal Expression Control
- 2026Beyond Words: Towards Effective Modeling of Non-Verbal Vocalizations in ASR
- 2026NVMOS: Non-Verbal Vocalization Quality Assessment in Speech
- 2026Affectron: Emotional Speech Synthesis with Affective and Contextually Aligned Nonverbal Vocalizations
- 2026Preference Optimization for Non-Verbal Vocalization Synthesis
- 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Models