# LAFMA: A Latent Flow Matching Model for Text-to-Audio Generation

arXiv: https://arxiv.org/abs/2406.08203 · HTML: https://mlsanity.com/paper/2406.08203/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Diffsound: Discrete Diffusion Model for Text-to-sound Generation](https://mlsanity.com/paper/2207.09983/) (2022)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech](https://mlsanity.com/paper/2207.06389/) (2022)
- [Audiobox: Unified Audio Generation with Natural Language Prompts](https://mlsanity.com/paper/2312.15821/) (2023)
- [Guided Flows for Generative Modeling and Decision Making](https://mlsanity.com/paper/2311.13443/) (2023)
- [Generative Pre-training for Speech with Flow Matching](https://mlsanity.com/paper/2310.16338/) (2023)
- [Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation](https://mlsanity.com/paper/2401.01044/) (2024)
- [CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model](https://mlsanity.com/paper/2305.06908/) (2023)
- [ReFlow-TTS: A Rectified Flow Model for High-fidelity Text-to-Speech](https://mlsanity.com/paper/2309.17056/) (2023)
- [MM-TTS: Multi-modal Prompt based Style Transfer for Expressive Text-to-Speech Synthesis](https://mlsanity.com/paper/2312.10687/) (2023)

## Built on by (most cited first, 24 of 24)

- [Taming Data and Transformers for Audio Generation](https://mlsanity.com/paper/2406.19388/) (2024)
- [SplitMeanFlow: Interval Splitting Consistency in Few-Step Generative Modeling](https://mlsanity.com/paper/2507.16884/) (2025)
- [MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flows](https://mlsanity.com/paper/2508.06098/) (2025)
- [Faster Inference of Flow-Based Generative Models via Improved Data-Noise Coupling](https://mlsanity.com/paper/2603.15279/) (2026)
- [UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities](https://mlsanity.com/paper/2509.24391/) (2025)
- [Local Flow Matching Generative Models](https://mlsanity.com/paper/2410.02548/) (2024)
- [YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controls](https://mlsanity.com/paper/2412.09168/) (2024)
- [Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversion](https://mlsanity.com/paper/2505.24291/) (2025)
- [Is There a Better Source Distribution than Gaussian? Exploring Source Distributions for Image Flow Matching](https://mlsanity.com/paper/2512.18184/) (2025)
- [SlimSpeech: Lightweight and Efficient Text-to-Speech with Slim Rectified Flow](https://mlsanity.com/paper/2504.07776/) (2025)
- [Generative Unordered Flow for Set-Structured Data Generation](https://mlsanity.com/paper/2501.17770/) (2025)
- [AudioTurbo: Fast Text-to-Audio Generation with Rectified Diffusion](https://mlsanity.com/paper/2505.22106/) (2025)
- [DreamAudio: Customized Text-to-Audio Generation with Diffusion Models](https://mlsanity.com/paper/2509.06027/) (2025)
- [Towards Stable and Structured Time Series Generation with Perturbation-Aware Flow Matching](https://mlsanity.com/paper/2511.14488/) (2025)
- [Preconditioned Flow Matching](https://mlsanity.com/paper/2603.02337/) (2026)
- [Geometry-Aware Image Flow Matching](https://mlsanity.com/paper/2605.25294/) (2026)
- [UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models](https://mlsanity.com/paper/2510.04593/) (2025)
- [GuitarFlow: Realistic Electric Guitar Synthesis From Tablatures via Flow Matching and Style Transfer](https://mlsanity.com/paper/2510.21872/) (2025)
- [AG-REPA: Causal Layer Selection for Representation Alignment in Audio Flow Matching](https://mlsanity.com/paper/2603.01006/) (2026)
- [UAT: Unified Audio-Text Diffusion for Audio Generation, Editing, and Captioning](https://mlsanity.com/paper/2606.04939/) (2026)
- [A Theory on Flow Matching with Neural Networks](https://mlsanity.com/paper/2606.10089/) (2026)
- [Investigating Group Relative Policy Optimization for Diffusion Transformer based Text-to-Audio Generation](https://mlsanity.com/paper/2603.01565/) (2026)
- [HoliDubber: Holistic Video Dubbing for Complex Acoustic Scenes via Text-Guided Audio Synthesis](https://mlsanity.com/paper/2606.09098/) (2026)
- [RIPPLE: Generating Multi-Channel Phase, Not Recovering It](https://mlsanity.com/paper/2607.27775/) (2026)

## Abstract (arXiv)

Recently, the application of diffusion models has facilitated the significant development of speech and audio generation. Nevertheless, the quality of samples generated by diffusion models still needs improvement. And the effectiveness of the method is accompanied by the extensive number of sampling steps, leading to an extended synthesis time necessary for generating high-quality audio. Previous Text-to-Audio (TTA) methods mostly used diffusion models in the latent space for audio generation. In this paper, we explore the integration of the Flow Matching (FM) model into the audio latent space for audio generation. The FM is an alternative simulation-free method that trains continuous normalization flows (CNF) based on regressing vector fields. We demonstrate that our model significantly enhances the quality of generated audio samples, achieving better performance than prior models. Moreover, it reduces the number of inference steps to ten steps almost without sacrificing performance.