mlsanityData to 7 Oct 2026

EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformer

arXiv 2409.10819 ↗First posted 17 Sep 2024eess.AS cs.SDcited by 30

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We introduce EzAudio, a text-to-audio (T2A) generation framework designed to produce high-quality, natural-sounding sound effects. Core designs include: (1) We propose EzAudio-DiT, an optimized Diffusion Transformer (DiT) designed for audio latent representations, improving convergence speed, as well as parameter and memory efficiency. (2) We apply a classifier-free guidance (CFG) rescaling technique to mitigate fidelity loss at higher CFG scores and enhancing prompt adherence without compromising audio quality. (3) We propose a synthetic caption generation strategy leveraging recent advances in audio understanding and LLMs to enhance T2A pretraining. We show that EzAudio, with its computationally efficient architecture and fast convergence, is a competitive open-source model that excels in both objective and subjective evaluations by delivering highly realistic listening experiences. Code, data, and pre-trained models are released at: https://haidog-yaqub.github.io/EzAudio-Page/.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 40 of its 44 references that are in the map. First posted on arXiv 17 Sep 2024.

Builds on

40 of 44 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Long-form music generation with latent diffusion16 Apr
  2. 2023Audiobox: Unified Audio Generation with Natural Language Prompts25 Dec
  3. 2023DPM-TSE: A Diffusion Probabilistic Model for Target Sound Extraction6 Oct
  4. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  5. 2023JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models9 Aug
  6. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation29 May
  7. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model24 Apr
  8. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models30 Jan
  9. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan

Other references · 31

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2022Scalable Diffusion Models with Transformerscited by 5,931
  6. 2022Classifier-Free Diffusion Guidancecited by 5,717
  7. 2016Layer Normalizationcited by 5,579
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
19 more
  1. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  2. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  3. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  4. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  5. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  6. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  7. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  8. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  9. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
  10. 2020Query-Key Normalization for Transformerscited by 266
  11. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  12. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 260
  13. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  14. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 151
  15. 2020Neural Networks Fail to Learn Periodic Functions and How to Fix Itcited by 97
  16. 2021The Benefit Of Temporally-Strong Labels In Audio Event Classificationcited by 83
  17. 2023Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learningcited by 32
  18. 2024Taming Data and Transformers for Audio Generationcited by 28
  19. 2024Improving Text-To-Audio Models with Synthetic Captionscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 15.

30 papers.

Sort:

By area:

  1. 2025KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generationcited by 24
  2. 2024A Survey of Sustainability in Large Language Models: Applications, Economics, and Challengescited by 18
  3. 2024ETTA: Elucidating the Design Space of Text-to-Audio Modelscited by 17
  4. 2025MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flowscited by 14
  5. 2025S2Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillationcited by 12
  6. 2025Guiding Audio Editing with Audio Language Modelcited by 11
  7. 2025SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipelinecited by 9
  8. 2025RFLAV: Rolling Flow matching for infinite Audio Video generationcited by 6
  9. 2025IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modelingcited by 6
  10. 2025UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalitiescited by 5
  11. 2025FlexSED: Towards Open-Vocabulary Sound Event Detectioncited by 4
  12. 2025SAO-Instruct: Free-form Audio Editing using Natural Language Instructionscited by 3
18 more
  1. 2026Audio ControlNet for Fine-Grained Audio Generation and Editingcited by 2
  2. 2026UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructionscited by 2
  3. 2025DegDiT: Controllable Audio Generation with Dynamic Event Graph Guided Diffusion Transformercited by 1
  4. 2025SynSonic: Augmenting Sound Event Detection through Text-to-Audio Diffusion ControlNet and Effective Sample Filteringcited by 1
  5. 2026ALM2Vec: Learning Audio Embeddings for Universal Audio Retrieval with Large Audio-Language Modelscited by 1
  6. 2025BridgeVoC: Revitalizing Neural Vocoder from a Restoration Perspectivecited by 0
  7. 2026SLD-L2S: Hierarchical Subspace Latent Diffusion for High-Fidelity Lip to Speech Synthesiscited by 0
  8. 2026PRoADS: Provably Secure and Robust Audio Diffusion Steganography with latent optimization and backward Euler Inversioncited by 0
  9. 2026DiT-Flow: Speech Enhancement Robust to Multiple Distortions based on Flow Matching in Latent Space and Diffusion Transformerscited by 0
  10. 2026LoSATok: Low-Dimensional Semantic-Acoustic Tokenizer for Cross-Domain Audio Understanding and Generationcited by 0
  11. 2026Inside the Latent Flow: Causal Deciphering of Attention Dynamics in Audio Separation Foundation Modelscited by 0
  12. 2026Adaptive Oscillatory Inductive Bias for Modeling Sharp Prosodic Dynamics in Diffusion-Based TTScited by 0
  13. 2026Elastic Time: Dynamic Frame Rate Bottlenecks for Neural Audio Codingcited by 0
  14. 2026ReGen: Hierarchical Multi-Prompt Representation Generation for Efficient Waveform Diffusion Modelscited by 0
  15. 2026Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Modelscited by 0
  16. 2026AudioScape-TTA: A Structured Soundscape Benchmark for Fine-Grained Text-to-Audio Evaluationcited by 0
  17. 2026VoxAudio: Vocalized Audio Synthesis via Multi-Reward Autoregressive Flow Matchingcited by 0
  18. 2026UNITE-AUDIO: Joint Learning of Continuous Tokenization and Latent Flow Matching for Text-to-Audio Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SwiftAudio: Data-Efficient Caption-Only Distillation for One-Step Text-to-Audio Diffusion-based Generation
  2. 2025Audio Palette: A Diffusion Transformer with Multi-Signal Conditioning for Controllable Foley Synthesis
  3. 2024AudioEditor: A Training-Free Diffusion-Based Audio Editing Framework
  4. 2025AudioTurbo: Fast Text-to-Audio Generation with Rectified Diffusion
  5. 2025DreamAudio: Customized Text-to-Audio Generation with Diffusion Models
  6. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation
  7. 2026AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generation
  8. 2025ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling