mlsanityData to 14 Oct 2026

ConsistencyTTA: Accelerating Diffusion-Based Text-to-Audio Generation with Consistency Distillation

arXiv 2309.10740 ↗First posted 19 Sep 2023cs.SD cs.LG cs.MMcited by 22

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Diffusion models are instrumental in text-to-audio (TTA) generation. Unfortunately, they suffer from slow inference due to an excessive number of queries to the underlying denoising network per generation. To address this bottleneck, we introduce ConsistencyTTA, a framework requiring only a single non-autoregressive network query, thereby accelerating TTA by hundreds of times. We achieve so by proposing "CFG-aware latent consistency model," which adapts consistency generation into a latent space and incorporates classifier-free guidance (CFG) into model training. Moreover, unlike diffusion models, ConsistencyTTA can be finetuned closed-loop with audio-space text-aware metrics, such as CLAP score, to further enhance the generations. Our objective and subjective evaluation on the AudioCaps dataset shows that compared to diffusion-based counterparts, ConsistencyTTA reduces inference computation by 400x while retaining generation quality and diversity.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 31 of its 39 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

31 of 39 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  2. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation29 May
  3. 2023CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model11 May
  4. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model24 Apr
  5. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models8 Feb
  6. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models30 Jan
  7. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  8. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul
  9. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis21 Apr

Other references · 22

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  10. 2023Consistency Modelscited by 1,822
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  12. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
10 more
  1. 2022On Distillation of Guided Diffusion Modelscited by 725
  2. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  3. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  4. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 700
  5. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  6. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 334
  7. 2022AudioGen: Textually Guided Audio Generationcited by 295
  8. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 246
  9. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  10. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 7; 2026: 3.

22 papers.

Sort:

By area:

  1. 2024ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulationcited by 57
  2. 2024DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generationcited by 26
  3. 2025KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generationcited by 24
  4. 2024AudioLCM: Text-to-Audio Generation with Latent Consistency Modelscited by 21
  5. 2024FlashAudio: Rectified Flows for Fast and High-Fidelity Text-to-Audio Generationcited by 19
  6. 2024Presto! Distilling Steps and Layers for Accelerating Music Generationcited by 16
  7. 2025Fast Text-to-Audio Generation with Adversarial Post-Trainingcited by 14
  8. 2025MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flowscited by 14
  9. 2024Diversity-Rewarded CFG Distillationcited by 9
  10. 2024MixEval-X: Any-to-Any Evaluations from Real-World Data Mixturescited by 6
  11. 2024Challenge on Sound Scene Synthesis: Evaluating Text-to-Audio Generationcited by 6
  12. 2024A Survey of Deep Learning Audio Generation Methodscited by 5
10 more
  1. 2025FlashSR: One-step Versatile Audio Super-resolution via Diffusion Distillationcited by 5
  2. 2025SoundReactor: Frame-level Online Video-to-Audio Generationcited by 5
  3. 2025DRAGON: Distributional Rewards Optimize Diffusion Generative Modelscited by 4
  4. 2024LiteFocus: Accelerated Diffusion Inference for Long Audio Synthesiscited by 3
  5. 2023QonFusion -- Quantum Approaches to Gaussian Random Variables: Applications in Stable Diffusion and Brownian Motioncited by 1
  6. 2026SwiftAudio: Data-Efficient Caption-Only Distillation for One-Step Text-to-Audio Diffusion-based Generationcited by 1
  7. 2024Let's Go Shopping (LGS) -- Web-Scale Image-Text Dataset for Visual Concept Understandingcited by 0
  8. 2025MIMII-Agent: Leveraging LLMs with Function Calling for Relative Evaluation of Anomalous Sound Detectioncited by 0
  9. 2026SoundWeaver: Semantic Warm-Starting for Text-to-Audio Diffusion Servingcited by 0
  10. 2026Real-Time Interactive Music Generation via Data-Free Streaming Consistency Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Fast Text-to-Audio Generation with One-Step Sampling via Energy-Scoring and Auxiliary Contextual Representation Distillation ↗
  2. 2024LAFMA: A Latent Flow Matching Model for Text-to-Audio Generation
  3. 2026Stable Audio 3
  4. 2024PTQ4ADM: Post-Training Quantization for Efficient Text Conditional Audio Diffusion Models
  5. 2025AudioTurbo: Fast Text-to-Audio Generation with Rectified Diffusion
  6. 2025IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
  7. 2025ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling
  8. 2026UAT: Unified Audio-Text Diffusion for Audio Generation, Editing, and Captioning