mlsanityData to 7 Oct 2026

Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Survey

arXiv 2412.06602 ↗First posted 9 Dec 2024cs.CL cs.AI cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Text-to-speech (TTS) has advanced from generating natural-sounding speech to enabling fine-grained control over attributes like emotion, timbre, and style. Driven by rising industrial demand and breakthroughs in deep learning, e.g., diffusion and large language models (LLMs), controllable TTS has become a rapidly growing research area. This survey provides the first comprehensive review of controllable TTS methods, from traditional control techniques to emerging approaches using natural language prompts. We categorize model architectures, control strategies, and feature representations, while also summarizing challenges, datasets, and evaluations in controllable TTS. This survey aims to guide researchers and practitioners by offering a clear taxonomy and highlighting future directions in this fast-evolving field. One can visit https://github.com/imxtx/awesome-controllabe-speech-synthesis for a comprehensive paper list and updates.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 130 of its 212 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

130 of 212 references.

References · 130

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  10. 2015Variational Inference with Normalizing Flowscited by 2,161
  11. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  12. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
38 more
  1. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  2. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  3. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  4. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  5. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  6. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  7. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  8. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  9. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  10. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  11. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  12. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  13. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  14. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  15. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  16. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  17. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  18. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  19. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  20. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  21. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  22. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  23. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  24. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  25. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  26. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  27. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  28. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
  29. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  30. 2018Adversarial Audio Synthesiscited by 222
  31. 2018Efficient Neural Audio Synthesiscited by 218
  32. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 189
  33. 2021A Survey on Neural Speech Synthesiscited by 182
  34. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 174
  35. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  36. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  37. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  38. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speechcited by 156

50 most cited of 130 All 130 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 20.

22 papers.

Sort:

By area:

  1. 2026EmoShift: Lightweight Activation Steering for Enhanced Emotion-Aware Speech Synthesiscited by 5
  2. 2026EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesiscited by 2
  3. 2026TiCo: Time-Controllable Spoken Dialogue Modelcited by 1
  4. 2026The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluationcited by 1
  5. 2026Bagpiper-TTS: Natural Language Guided Universal Speech Synthesiscited by 1
  6. 2025TokenChain: A Discrete Speech Chain via Semantic Token Modelingcited by 0
  7. 2025Cross-modal Consistency Guidance for Robust Emotion Control in Auto-Regressive TTS Modelscited by 0
  8. 2026SpeakerSleuth: Can Large Audio-Language Models Judge Speaker Consistency across Multi-turn Dialogues?cited by 0
  9. 2026LJ-Spoof: A Generatively Varied Corpus for Audio Anti-Spoofing and Synthesis Source Tracingcited by 0
  10. 2026On the Emotion Understanding of Synthesized Speechcited by 0
  11. 2026Controllable Singing Style Conversion with Boundary-Aware Information Bottleneckcited by 0
  12. 2026Anonymization, Not Elimination: Utility-Preserved Speech Anonymizationcited by 0
10 more
  1. 2026A Generalized Formalism of Auto-Regressive Decoding for Speech Processingcited by 0
  2. 2026Adaptive Oscillatory Inductive Bias for Modeling Sharp Prosodic Dynamics in Diffusion-Based TTScited by 0
  3. 2026VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptationcited by 0
  4. 2026Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensionscited by 0
  5. 2026Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correctioncited by 0
  6. 2026Ready to Speak: Aligning LLMs for TTS-Friendly Text Generationcited by 0
  7. 2026Hearing the Whispers: Black-Box Membership Inference Attacks on Finetuned TTS Modelscited by 0
  8. 2026Continuous-Time Acoustic Modelling with Neural Controlled Differential Equationscited by 0
  9. 2026COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoningcited by 0
  10. 2026SEmoEdit: Probing and Harnessing the Editability of Pre-trained Speech Flowscited by 0