# Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis

arXiv: https://arxiv.org/abs/2307.07218 · HTML: https://mlsanity.com/paper/2307.07218/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2023.

## Built on by (most cited first, 30 of 41)

- [XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model](https://mlsanity.com/paper/2406.04904/) (2024)
- [WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling](https://mlsanity.com/paper/2408.16532/) (2024)
- [VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild](https://mlsanity.com/paper/2403.16973/) (2024)
- [IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2506.21619/) (2025)
- [DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation](https://mlsanity.com/paper/2502.03930/) (2025)
- [MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis](https://mlsanity.com/paper/2502.18924/) (2025)
- [HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesis](https://mlsanity.com/paper/2311.12454/) (2023)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors](https://mlsanity.com/paper/2406.11427/) (2024)
- [LeVo: High-Quality Song Generation with Multi-Preference Alignment](https://mlsanity.com/paper/2506.07520/) (2025)
- [FlashSpeech: Efficient Zero-Shot Speech Synthesis](https://mlsanity.com/paper/2404.14700/) (2024)
- [SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models](https://mlsanity.com/paper/2408.13893/) (2024)
- [WavReward: Spoken Dialogue Models With Generalist Reward Evaluators](https://mlsanity.com/paper/2505.09558/) (2025)
- [PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generation](https://mlsanity.com/paper/2408.07547/) (2024)
- [SongCreator: Lyrics-based Universal Song Generation](https://mlsanity.com/paper/2409.06029/) (2024)
- [Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like](https://mlsanity.com/paper/2402.07383/) (2024)
- [Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesis](https://mlsanity.com/paper/2504.10352/) (2025)
- [Mitigating Unauthorized Speech Synthesis for Voice Protection](https://mlsanity.com/paper/2410.20742/) (2024)
- [SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesis](https://mlsanity.com/paper/2504.09839/) (2025)
- [Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversion](https://mlsanity.com/paper/2505.24291/) (2025)
- [Versatile Framework for Song Generation with Prompt-based Control](https://mlsanity.com/paper/2504.19062/) (2025)
- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [RASMALAI: Resources for Adaptive Speech Modeling in Indian Languages with Accents and Intonations](https://mlsanity.com/paper/2505.18609/) (2025)
- [Conan: A Chunkwise Online Network for Zero-Shot Adaptive Voice Conversion](https://mlsanity.com/paper/2507.14534/) (2025)
- [SupertonicTTS: Towards Highly Efficient and Streamlined Text-to-Speech System](https://mlsanity.com/paper/2503.23108/) (2025)
- [Efficient Interleaved Speech Modeling through Knowledge Distillation](https://mlsanity.com/paper/2506.23670/) (2025)
- [Investigating Disentanglement in a Phoneme-level Speech Codec for Prosody Modeling](https://mlsanity.com/paper/2409.08664/) (2024)
- [Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modeling](https://mlsanity.com/paper/2410.01350/) (2024)
- [MultiVerse: Efficient and Expressive Zero-Shot Multi-Task Text-to-Speech](https://mlsanity.com/paper/2410.03192/) (2024)
- [Lina-Speech: Gated Linear Attention and Initial-State Tuning for Multi-Sample Prompting Text-To-Speech Synthesis](https://mlsanity.com/paper/2410.23320/) (2024)

## Abstract (arXiv)

Zero-shot text-to-speech (TTS) aims to synthesize voices with unseen speech prompts, which significantly reduces the data and computation requirements for voice cloning by skipping the fine-tuning process. However, the prompting mechanisms of zero-shot TTS still face challenges in the following aspects: 1) previous works of zero-shot TTS are typically trained with single-sentence prompts, which significantly restricts their performance when the data is relatively sufficient during the inference stage. 2) The prosodic information in prompts is highly coupled with timbre, making it untransferable to each other. This paper introduces Mega-TTS 2, a generic prompting mechanism for zero-shot TTS, to tackle the aforementioned challenges. Specifically, we design a powerful acoustic autoencoder that separately encodes the prosody and timbre information into the compressed latent space while providing high-quality reconstructions. Then, we propose a multi-reference timbre encoder and a prosody latent language model (P-LLM) to extract useful information from multi-sentence prompts. We further leverage the probabilities derived from multiple P-LLM outputs to produce transferable and controllable prosody. Experimental results demonstrate that Mega-TTS 2 could not only synthesize identity-preserving speech with a short prompt of an unseen speaker from arbitrary sources but consistently outperform the fine-tuning method when the volume of data ranges from 10 seconds to 5 minutes. Furthermore, our method enables to transfer various speaking styles to the target timbre in a fine-grained and controlled manner. Audio samples can be found in https://boostprompt.github.io/boostprompt/.