# F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

arXiv: https://arxiv.org/abs/2410.06885 · HTML: https://mlsanity.com/paper/2410.06885/

## Key facts

- Cited by 361 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 68 of its 76 references that are in the map.
- First posted on arXiv 9 Oct 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 68)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [Analyzing and Improving the Training Dynamics of Diffusion Models](https://mlsanity.com/paper/2312.02696/) (2023)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens](https://mlsanity.com/paper/2407.05407/) (2024)
- [Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation](https://mlsanity.com/paper/2303.00848/) (2023)
- [Seed-TTS: A Family of High-Quality Versatile Speech Generation Models](https://mlsanity.com/paper/2406.02430/) (2024)

## Built on by (most cited first, 30 of 361)

- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [Qwen3.5-Omni Technical Report](https://mlsanity.com/paper/2604.15804/) (2026)
- [CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training](https://mlsanity.com/paper/2505.17589/) (2025)
- [Qwen3-TTS Technical Report](https://mlsanity.com/paper/2601.15621/) (2026)
- [Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens](https://mlsanity.com/paper/2503.01710/) (2025)
- [WavChat: A Survey of Spoken Dialogue Models](https://mlsanity.com/paper/2411.13577/) (2024)
- [IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2506.21619/) (2025)
- [Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation](https://mlsanity.com/paper/2510.01284/) (2025)
- [Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis](https://mlsanity.com/paper/2502.04128/) (2025)
- [DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation](https://mlsanity.com/paper/2502.03930/) (2025)
- [InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions](https://mlsanity.com/paper/2412.09596/) (2024)
- [DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusion](https://mlsanity.com/paper/2503.01183/) (2025)
- [Zero-shot Voice Conversion with Diffusion Transformers](https://mlsanity.com/paper/2411.09943/) (2024)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis](https://mlsanity.com/paper/2502.18924/) (2025)
- [UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions](https://mlsanity.com/paper/2511.03334/) (2025)
- [MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation](https://mlsanity.com/paper/2411.15262/) (2024)
- [URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models](https://mlsanity.com/paper/2502.17810/) (2025)
- [ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching](https://mlsanity.com/paper/2506.13053/) (2025)
- [SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generation](https://mlsanity.com/paper/2507.09862/) (2025)
- [VibeVoice Technical Report](https://mlsanity.com/paper/2508.19205/) (2025)
- [IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System](https://mlsanity.com/paper/2502.05512/) (2025)
- [Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance](https://mlsanity.com/paper/2502.05236/) (2025)
- [Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling](https://mlsanity.com/paper/2509.08753/) (2025)
- [OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models](https://mlsanity.com/paper/2604.00688/) (2026)
- [F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimization](https://mlsanity.com/paper/2504.02407/) (2025)
- [Paper2Video: Automatic Video Generation from Scientific Papers](https://mlsanity.com/paper/2510.05096/) (2025)
- [Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2412.06602/) (2024)

## Abstract (arXiv)

This paper introduces F5-TTS, a fully non-autoregressive text-to-speech system based on flow matching with Diffusion Transformer (DiT). Without requiring complex designs such as duration model, text encoder, and phoneme alignment, the text input is simply padded with filler tokens to the same length as input speech, and then the denoising is performed for speech generation, which was originally proved feasible by E2 TTS. However, the original design of E2 TTS makes it hard to follow due to its slow convergence and low robustness. To address these issues, we first model the input with ConvNeXt to refine the text representation, making it easy to align with the speech. We further propose an inference-time Sway Sampling strategy, which significantly improves our model's performance and efficiency. This sampling strategy for flow step can be easily applied to existing flow matching based models without retraining. Our design allows faster training and achieves an inference RTF of 0.15, which is greatly improved compared to state-of-the-art diffusion-based TTS models. Trained on a public 100K hours multilingual dataset, our F5-TTS exhibits highly natural and expressive zero-shot ability, seamless code-switching capability, and speed control efficiency. We have released all codes and checkpoints to promote community development, at https://SWivid.github.io/F5-TTS/.