Improving Audio Codec-based Zero-Shot Text-to-Speech Synthesis with Multi-Modal Context and Large Language Model
Not on the map. Why?
Abstract (arXiv)
Recent advances in large language models (LLMs) and development of audio codecs greatly propel the zero-shot TTS. They can synthesize personalized speech with only a 3-second speech of an unseen speaker as acoustic prompt. However, they only support short speech prompts and cannot leverage longer context information, as required in audiobook and conversational TTS scenarios. In this paper, we introduce a novel audio codec-based TTS model to adapt context features with multiple enhancements. Inspired by the success of Qformer, we propose a multi-modal context-enhanced Qformer (MMCE-Qformer) to utilize additional multi-modal context information. Besides, we adapt a pretrained LLM to leverage its understanding ability to predict semantic tokens, and use a SoundStorm to generate acoustic tokens thereby enhancing audio quality and speaker similarity. The extensive objective and subjective evaluations show that our proposed method outperforms baselines across various context TTS scenarios.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 29 of its 33 references that are in the map. First posted on arXiv 6 Jun 2024.
Builds on
29 of 33 references.
References · 29
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
- 2022MaskGIT: Masked Generative Image Transformercited by 1,082
- 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
- 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
- 2022High Fidelity Neural Audio Compressioncited by 673
17 more
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
- 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
- 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
- 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
- 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
- 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
- 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
- 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
- 2019Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddingscited by 73
- 2023Better speech synthesis through scalingcited by 63
- 2021SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Modelcited by 56
- 2020Conversational End-to-End TTS for Voice Agentcited by 37
- 2023CONCSS: Contrastive-based Context Comprehension for Dialogue-appropriate Prosody in Conversational Speech Synthesiscited by 8
- 2022Improving Speech Prosody of Audiobook Text-to-Speech Synthesis with Acoustic and Textual Contextscited by 6
- 2023M2-CTTS: End-to-End Multi-scale Multi-modal Conversational Text-to-Speech Synthesiscited by 6
- 2022ParaTTS: Learning Linguistic and Prosodic Cross-sentence Information in Paragraph-based TTScited by 5
- 2023MSStyleTTS: Multi-Scale Style Modeling with Hierarchical Context Information for Expressive Speech Synthesiscited by 5
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.