# ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching

arXiv: https://arxiv.org/abs/2506.13053 · HTML: https://mlsanity.com/paper/2506.13053/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 51 of its 57 references that are in the map.
- First posted on arXiv 16 Jun 2025.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching](https://mlsanity.com/paper/2410.06885/) (2024)
- [CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens](https://mlsanity.com/paper/2407.05407/) (2024)
- [Seed-TTS: A Family of High-Quality Versatile Speech Generation Models](https://mlsanity.com/paper/2406.02430/) (2024)
- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation](https://mlsanity.com/paper/2407.05361/) (2024)
- [FreeU: Free Lunch in Diffusion U-Net](https://mlsanity.com/paper/2309.11497/) (2023)
- [MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer](https://mlsanity.com/paper/2409.00750/) (2024)
- [Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis](https://mlsanity.com/paper/2306.00814/) (2023)

## Built on by (most cited first, 30 of 30)

- [OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models](https://mlsanity.com/paper/2604.00688/) (2026)
- [ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching](https://mlsanity.com/paper/2507.09318/) (2025)
- [MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloning](https://mlsanity.com/paper/2601.01568/) (2026)
- [M3-TTS: Multi-modal DiT Alignment & Mel-latent for Zero-shot High-fidelity Speech Synthesis](https://mlsanity.com/paper/2512.04720/) (2025)
- [IndexTTS 2.5 Technical Report](https://mlsanity.com/paper/2601.03888/) (2026)
- [Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation](https://mlsanity.com/paper/2605.08729/) (2026)
- [VoXtream: Full-Stream Text-to-Speech with Extremely Low Latency](https://mlsanity.com/paper/2509.15969/) (2025)
- [InstructAudio: Unified speech and music generation with natural language instruction](https://mlsanity.com/paper/2511.18487/) (2025)
- [Flow2GAN: Hybrid Flow Matching and GAN with Multi-Resolution Network for Few-step High-Fidelity Audio Generation](https://mlsanity.com/paper/2512.23278/) (2025)
- [MMControl: Unified Multi-Modal Control for Joint Audio-Video Generation](https://mlsanity.com/paper/2604.19679/) (2026)
- [Habibi: Laying the Open-Source Foundation of Unified-Dialectal Arabic Speech Synthesis](https://mlsanity.com/paper/2601.13802/) (2026)
- [VoXtream2: Full-stream TTS with dynamic speaking rate control](https://mlsanity.com/paper/2603.13518/) (2026)
- [LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space](https://mlsanity.com/paper/2603.29339/) (2026)
- [Chatterbox-Flash: Prior-Calibrated Block Diffusion for Streaming Zero-Shot TTS](https://mlsanity.com/paper/2605.30748/) (2026)
- [UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions](https://mlsanity.com/paper/2604.22209/) (2026)
- [DiTReducio: A Training-Free Acceleration for DiT-Based TTS via Progressive Calibration](https://mlsanity.com/paper/2509.09748/) (2025)
- [WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modeling](https://mlsanity.com/paper/2606.03455/) (2026)
- [SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS](https://mlsanity.com/paper/2512.05126/) (2025)
- [What Counts as Real? Speech Restoration and Voice Quality Conversion Pose New Challenges to Deepfake Detection](https://mlsanity.com/paper/2603.14033/) (2026)
- [Conditioned in Training, Dispensable at Inference: Typological Conditioning for Multilingual Speech Translation](https://mlsanity.com/paper/2605.16026/) (2026)
- [Towards Unified Song Generation and Singing Voice Conversion with Accompaniment Co-Generation](https://mlsanity.com/paper/2606.07015/) (2026)
- [End-to-End Training for Discrete Token LLM based TTS System](https://mlsanity.com/paper/2606.09234/) (2026)
- [Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors](https://mlsanity.com/paper/2606.19325/) (2026)
- [ReGen: Hierarchical Multi-Prompt Representation Generation for Efficient Waveform Diffusion Models](https://mlsanity.com/paper/2607.09134/) (2026)
- [FreyaTTS: A Compact Tokenizer-Free Flow-Matching Transformer for Turkish-First Speech Synthesis](https://mlsanity.com/paper/2607.09530/) (2026)
- [Faster IndexTTS-2: Accelerating and Streaming Autoregressive Zero-Shot Text-to-Speech Synthesis on GPUs](https://mlsanity.com/paper/2607.21042/) (2026)
- [Zero-Shot Face-to-Speech Synthesis via Latent Space Adaptation of a Style-Diffusion TTS Model](https://mlsanity.com/paper/2607.26742/) (2026)
- [CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents](https://mlsanity.com/paper/2608.08638/) (2026)
- [Not Quite My Tempo: Voice Activity-aware Speech Synthesis for Lip-Synchronous Dubbing](https://mlsanity.com/paper/2609.26486/) (2026)
- [ReaFlow-TTS: Realization-Conditioned Flow Matching for High-Quality and Controllable Speech Synthesis](https://mlsanity.com/paper/2609.28906/) (2026)

## Abstract (arXiv)

Existing large-scale zero-shot text-to-speech (TTS) models deliver high speech quality but suffer from slow inference speeds due to massive parameters. To address this issue, this paper introduces ZipVoice, a high-quality flow-matching-based zero-shot TTS model with a compact model size and fast inference speed. Key designs include: 1) a Zipformer-based vector field estimator to maintain adequate modeling capabilities under constrained size; 2) Average upsampling-based initial speech-text alignment and Zipformer-based text encoder to improve speech intelligibility; 3) A flow distillation method to reduce sampling steps and eliminate the inference overhead associated with classifier-free guidance. Experiments on 100k hours multilingual datasets show that ZipVoice matches state-of-the-art models in speech quality, while being 3 times smaller and up to 30 times faster than a DiT-based flow-matching baseline. Codes, model checkpoints and demo samples are publicly available at https://github.com/k2-fsa/ZipVoice.