mlsanityData to 14 Oct 2026

CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech

arXiv 2506.02863 ↗First posted 3 Jun 2025eess.AS cs.AI cs.SDcited by 11

Not on the map. Why?

Abstract (arXiv)

Recent advancements in generative artificial intelligence have significantly transformed the field of style-captioned text-to-speech synthesis (CapTTS). However, adapting CapTTS to real-world applications remains challenging due to the lack of standardized, comprehensive datasets and limited research on downstream tasks built upon CapTTS. To address these gaps, we introduce CapSpeech, a new benchmark designed for a series of CapTTS-related tasks, including style-captioned text-to-speech synthesis with sound events (CapTTS-SE), accent-captioned TTS (AccCapTTS), emotion-captioned TTS (EmoCapTTS), and text-to-speech synthesis for chat agent (AgentTTS). CapSpeech comprises over 10 million machine-annotated audio-caption pairs and nearly 0.36 million human-annotated audio-caption pairs. In addition, we introduce two new datasets collected and recorded by a professional voice actor and experienced audio engineers, specifically for the AgentTTS and CapTTS-SE tasks. Alongside the datasets, we conduct comprehensive experiments using both autoregressive and non-autoregressive models on CapSpeech. Our results demonstrate high-fidelity and highly intelligible speech synthesis across a diverse range of speaking styles. To the best of our knowledge, CapSpeech is the largest available dataset offering comprehensive annotations for CapTTS-related tasks. The experiments and findings further provide valuable insights into the challenges of developing CapTTS systems.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 74 of its 87 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

74 of 87 references.

References · 74

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  8. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  10. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  11. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  12. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
38 more
  1. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  2. 2024Qwen2-Audio Technical Reportcited by 560
  3. 2023Simple and Controllable Music Generationcited by 488
  4. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  5. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  6. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  7. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  8. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  9. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  10. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  11. 2020Query-Key Normalization for Transformerscited by 280
  12. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  13. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  14. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  15. 2025Kimi-Audio Technical Reportcited by 222
  16. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  17. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 162
  18. 2021ASVspoof 2021: accelerating progress in spoofed and deepfake speech detectioncited by 161
  19. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  20. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  21. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  22. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  23. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 110
  24. 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 101
  25. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 98
  26. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 92
  27. 2024The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speechcited by 83
  28. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 70
  29. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  30. 2024EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberationcited by 64
  31. 2020One-class Learning Towards Synthetic Voice Spoofing Detectioncited by 61
  32. 2023TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudiocited by 59
  33. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
  34. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 48
  35. 2022EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidancecited by 37
  36. 2023The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASRcited by 36
  37. 2023PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptionscited by 34
  38. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 34

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2025MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speechcited by 6
  2. 2026NV-Bench: Benchmark of Nonverbal Vocalization Synthesis for Expressive Text-to-Speech Generationcited by 6
  3. 2025Spoken DialogSum: An Emotion-Rich Conversational Dataset for Spoken Dialogue Summarizationcited by 2
  4. 2026WESR: Scaling and Evaluating Word-level Event-Speech Recognitioncited by 1
  5. 2026TRACE: Temporal Relationship-Aware Conversational Entrainment Detection in Dyadic Speechcited by 1
  6. 2026VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentationcited by 1
  7. 2025BatonVoice: An Operationalist Framework for Enhancing Controllable Speech Synthesis with Linguistic Intelligence from LLMscited by 0
  8. 2026Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Modelscited by 0
  9. 2026How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speechcited by 0
  10. 2026Emputation: Identification-Guided Neural Imputation Frameworkcited by 0
  11. 2026Modeling, Scaling, and Decoding: Optimizing Controllable Speech Generation with Nonverbal Vocalizationscited by 0