mlsanityData to 19 Oct 2026

Affectron: Emotional Speech Synthesis with Affective and Contextually Aligned Nonverbal Vocalizations

arXiv 2603.14432 ↗First posted 15 Mar 2026cs.SDcited by 0

On the map: Speech and audio language models → Emotional speech synthesis and recognition

Abstract (arXiv)

Nonverbal vocalizations (NVs), such as laughter and sighs, are central to the expression of affective cues in emotional speech synthesis. However, learning diverse and contextually aligned NVs remains challenging in open settings due to limited NV data and the lack of explicit supervision. Motivated by this challenge, we propose Affectron as a framework for affective and contextually aligned NV generation. Built on a small-scale open and decoupled corpus, Affectron introduces an NV-augmented training strategy that expands the distribution of NV types and insertion locations. We further incorporate NV structural masking into a speech backbone pre-trained on purely verbal speech to enable diverse and natural NV synthesis. Experimental results demonstrate that Affectron produces more expressive and diverse NVs than baseline systems while preserving the naturalness of the verbal speech stream.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Emotional speech synthesis and recognition, one of 11 directions in Speech and audio language models. Builds on 33 of its 66 references that are in the map. First posted on arXiv 15 Mar 2026.

Builds on

33 of 66 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech Modeling with Paralinguistic Vocalizations6 Aug
  2. 2025NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech17 Jul
  3. 2025DiEmo-TTS: Disentangled Emotion Representations via Self-Supervised Distillation for Cross-Speaker Emotion Transfer in Text-to-Speech26 May
  4. 2024EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vector4 Nov
  5. 2024Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech17 Jul
  6. 2024Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like12 Feb
  7. 2023JVNV: A Corpus of Japanese Emotional Speech with Verbal Content and Nonverbal Expressions9 Oct

Other references · 26

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,661
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 4,385
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,587
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,113
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,762
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 2,135
  7. 2018Hierarchical Neural Story Generationcited by 1,209
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 1,007
  9. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 497
  10. 2022Efficient Training of Language Models to Fill in the Middlecited by 265
  11. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 263
  12. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 215
14 more
  1. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 169
  2. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 153
  3. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 146
  4. 2020Enabling Language Models to Fill in the Blankscited by 123
  5. 2024EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberationcited by 98
  6. 2022Emotion Intensity and its Control for Emotional Voice Conversioncited by 41
  7. 2024EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speechcited by 32
  8. 2025A Scalable Pipeline for Enabling Non-Verbal Speech Generation and Understandingcited by 16
  9. 2024Effective Pre-Training of Audio Transformers for Sound Event Detectioncited by 14
  10. 2023SponTTS: modeling and transferring spontaneous style for TTScited by 11
  11. 2023Laughter Synthesis using Pseudo Phonetic Tokens with a Large-scale In-the-wild Laughter Corpuscited by 8
  12. 2023JNV Corpus: A Corpus of Japanese Nonverbal Vocalizations with Diverse Phrases and Emotionscited by 5
  13. 2024EmoReg: Directional Latent Vector Modeling for Emotional Intensity Regularization in Diffusion-based Voice Conversioncited by 3
  14. 2025EmoSphere-SER: Enhancing Speech Emotion Recognition Through Spherical Representation with Auxiliary Classificationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Toward Natural Emotional Text-To-Speech System with Fine-Grained Non-Verbal Expression Control
  2. 2026Beyond Words: Towards Effective Modeling of Non-Verbal Vocalizations in ASR
  3. 2026Preference Optimization for Non-Verbal Vocalization Synthesis
  4. 2026NV-Bench: Benchmark of Nonverbal Vocalization Synthesis for Expressive Text-to-Speech Generation
  5. 2026NVMOS: Non-Verbal Vocalization Quality Assessment in Speech
  6. 2023AffectEcho: Speaker Independent and Language-Agnostic Emotion and Affect Transfer for Speech Synthesis
  7. 2026TRACE-EVC: Text-Guided Relative Affective Control for Zero-Shot Emotional Voice Conversion ↗
  8. 2026Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue