mlsanityData to 14 Oct 2026

F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

arXiv 2410.06885 ↗First posted 9 Oct 2024eess.AS cs.SDcited by 361

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

This paper introduces F5-TTS, a fully non-autoregressive text-to-speech system based on flow matching with Diffusion Transformer (DiT). Without requiring complex designs such as duration model, text encoder, and phoneme alignment, the text input is simply padded with filler tokens to the same length as input speech, and then the denoising is performed for speech generation, which was originally proved feasible by E2 TTS. However, the original design of E2 TTS makes it hard to follow due to its slow convergence and low robustness. To address these issues, we first model the input with ConvNeXt to refine the text representation, making it easy to align with the speech. We further propose an inference-time Sway Sampling strategy, which significantly improves our model's performance and efficiency. This sampling strategy for flow step can be easily applied to existing flow matching based models without retraining. Our design allows faster training and achieves an inference RTF of 0.15, which is greatly improved compared to state-of-the-art diffusion-based TTS models. Trained on a public 100K hours multilingual dataset, our F5-TTS exhibits highly natural and expressive zero-shot ability, seamless code-switching capability, and speed control efficiency. We have released all codes and checkpoints to promote community development, at https://SWivid.github.io/F5-TTS/.

Key facts. Cited by 361 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 68 of its 76 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

68 of 76 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024E1 TTS: Simple and Fast Non-Autoregressive TTS14 Sep
  2. 2024FLUX that Plays Music1 Sep
  3. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors17 Jun
  4. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesis8 Jun
  5. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  6. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speech2 Nov
  7. 2023VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matching10 Sep
  8. 2023Matcha-TTS: A fast TTS architecture with conditional flow matching6 Sep
  9. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models13 Jun
  10. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers18 Apr
  11. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May

Other references · 57

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2022Flow Matching for Generative Modelingcited by 5,702
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  11. 2022A ConvNet for the 2020scited by 3,579
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
38 more
  1. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  3. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  4. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  5. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  6. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  7. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 712
  8. 2022High Fidelity Neural Audio Compressioncited by 687
  9. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  10. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  11. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  13. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  14. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  15. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  16. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentationcited by 304
  17. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  18. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  19. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  20. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  21. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  22. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  23. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 153
  24. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 152
  25. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  26. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  27. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  28. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 97
  29. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 95
  30. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  31. 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 85
  32. 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 78
  33. 2024FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applicationscited by 68
  34. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  35. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  36. 2023LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Modelscited by 49
  37. 2023AudioDec: An Open-source Streaming High-fidelity Neural Audio Codeccited by 47
  38. 2022A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editingcited by 40

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 158; 2026: 192.

361 papers.

Sort:

By area:

  1. 2025Qwen2.5-Omni Technical Reportcited by 736
  2. 2025Qwen3-Omni Technical Reportcited by 497
  3. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  4. 2026Qwen3.5-Omni Technical Reportcited by 174
  5. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  6. 2026Qwen3-TTS Technical Reportcited by 118
  7. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 113
  8. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  9. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
  10. 2025Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generationcited by 83
  11. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 70
  12. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generationcited by 57
38 more
  1. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  2. 2025DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusioncited by 46
  3. 2024Zero-shot Voice Conversion with Diffusion Transformerscited by 43
  4. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  5. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 41
  6. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 38
  7. 2024MovieBench: A Hierarchical Movie Level Dataset for Long Video Generationcited by 36
  8. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Modelscited by 30
  9. 2025ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matchingcited by 30
  10. 2025SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generationcited by 30
  11. 2025VibeVoice Technical Reportcited by 29
  12. 2025IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech Systemcited by 28
  13. 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidancecited by 27
  14. 2025Streaming Sequence-to-Sequence Learning with Delayed Streams Modelingcited by 27
  15. 2026OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Modelscited by 25
  16. 2025F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimizationcited by 24
  17. 2025Paper2Video: Automatic Video Generation from Scientific Paperscited by 24
  18. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 23
  19. 2025Celeb-DF++: A Large-scale Challenging Video DeepFake Benchmark for Generalizable Forensicscited by 21
  20. 2025Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generationcited by 20
  21. 2025FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systemscited by 20
  22. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  23. 2025FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matchingcited by 17
  24. 2025InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systemscited by 17
  25. 2025Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representationcited by 17
  26. 2025S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Modelscited by 16
  27. 2025JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Modelscited by 16
  28. 2025Differentiable Reward Optimization for LLM based TTS systemcited by 15
  29. 2025SpeechJudge: Towards Human-Level Judgment for Speech Naturalnesscited by 15
  30. 2025QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptionscited by 13
  31. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 13
  32. 2025AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generationcited by 12
  33. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editingcited by 12
  34. 2025CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic Audiovisual Narrative Processingcited by 11
  35. 2025CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matchingcited by 11
  36. 2025CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speechcited by 11
  37. 2025Transition Matching: Scalable and Flexible Generative Modelingcited by 11
  38. 2025ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modelingcited by 11

50 most cited of 361 All 361 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025RapFlow-TTS: Rapid and High-Fidelity Text-to-Speech with Improved Consistency Flow Matching
  2. 2023ReFlow-TTS: A Rectified Flow Model for High-fidelity Text-to-Speech
  3. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
  4. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models
  5. 2025Discrete Diffusion for Generative Modeling of Text-Aligned Speech Tokens
  6. 2024Boosting Diffusion Model for Spectrogram Up-sampling in Text-to-speech: An Empirical Study
  7. 2025Next Tokens Denoising for Speech Synthesis
  8. 2024CM-TTS: Enhancing Real Time Text-to-Speech Synthesis Efficiency through Weighted Samplers and Consistency Models