mlsanityData to 14 Oct 2026

Description-based Controllable Text-to-Speech with Cross-Lingual Voice Control

arXiv 2409.17452 ↗First posted 26 Sep 2024eess.AS cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

We propose a novel description-based controllable text-to-speech (TTS) method with cross-lingual control capability. To address the lack of audio-description paired data in the target language, we combine a TTS model trained on the target language with a description control model trained on another language, which maps input text descriptions to the conditional features of the TTS model. These two models share disentangled timbre and style representations based on self-supervised learning (SSL), allowing for disentangled voice control, such as controlling speaking styles while retaining the original timbre. Furthermore, because the SSL-based timbre and style representations are language-agnostic, combining the TTS and description control models while sharing the same embedding space effectively enables cross-lingual control of voice characteristics. Experiments on English and Japanese TTS demonstrate that our method achieves high naturalness and controllability for both languages, even though no Japanese audio-description pairs are used.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 32 of its 40 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

32 of 40 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  8. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  9. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  10. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  11. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  12. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 152
20 more
  1. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  2. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  3. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 110
  4. 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 101
  5. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 92
  6. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  7. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
  8. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 74
  9. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
  10. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 48
  11. 2022NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesiscited by 38
  12. 2023PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptionscited by 34
  13. 2023PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptionscited by 25
  14. 2015Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problemscited by 23
  15. 2024VoxInstruct: Expressive Human Instruction-to-Speech Generation with Unified Multilingual Codec Language Modellingcited by 21
  16. 2024LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioningcited by 15
  17. 2023Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation modelcited by 10
  18. 2023Coco-Nut: Corpus of Japanese Utterance and Voice Characteristics Description for Prompt-based Controlcited by 7
  19. 2022Period VITS: Variational Inference with Explicit Pitch Modeling for End-to-end Emotional Speech Synthesiscited by 6
  20. 2023A Quantitative Approach to Understand Self-Supervised Models as Cross-lingual Feature Extractorscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 23
  2. 2025Scaling Rich Style-Prompted Text-to-Speech Datasetscited by 19
  3. 2024Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologiescited by 5
  4. 2026EmoInstruct-TTS: Dual-Path Instruction-Guided Emotional Speech Synthesiscited by 0