# Leveraging Pre-trained AudioLDM for Sound Generation: A Benchmark Study

arXiv: https://arxiv.org/abs/2303.03857 · HTML: https://mlsanity.com/paper/2303.03857/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 33 references that are in the map.
- First posted on arXiv 7 Mar 2023.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models](https://mlsanity.com/paper/2301.12661/) (2023)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [Diffsound: Discrete Diffusion Model for Text-to-sound Generation](https://mlsanity.com/paper/2207.09983/) (2022)
- [Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms](https://mlsanity.com/paper/1812.08466/) (2018)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [Taming Visually Guided Sound Generation](https://mlsanity.com/paper/2110.08791/) (2021)
- [Separate What You Describe: Language-Queried Audio Source Separation](https://mlsanity.com/paper/2203.15147/) (2022)
- [Neural Vocoder is All You Need for Speech Super-resolution](https://mlsanity.com/paper/2203.14941/) (2022)
- [On Metric Learning for Audio-Text Cross-Modal Retrieval](https://mlsanity.com/paper/2203.15537/) (2022)
- [Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning](https://mlsanity.com/paper/2107.09998/) (2021)
- [AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos with Deep Learning](https://mlsanity.com/paper/2002.10981/) (2020)
- [Leveraging Pre-trained BERT for Audio Captioning](https://mlsanity.com/paper/2203.02838/) (2022)
- [Adapting Pre-trained Vision Transformers from 2D to 3D through Weight Inflation Improves Medical Image Segmentation](https://mlsanity.com/paper/2302.04303/) (2023)
- [Self-Supervised In-Domain Representation Learning for Remote Sensing Image Scene Classification](https://mlsanity.com/paper/2302.01793/) (2023)
- [Few-Shot Table-to-Text Generation with Prompt Planning and Knowledge Memorization](https://mlsanity.com/paper/2302.04415/) (2023)

## Built on by (most cited first, 10 of 10)

- [V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models](https://mlsanity.com/paper/2308.09300/) (2023)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [WavJourney: Compositional Audio Creation with Large Language Models](https://mlsanity.com/paper/2307.14335/) (2023)
- [Retrieval-Augmented Text-to-Audio Generation](https://mlsanity.com/paper/2309.08051/) (2023)
- [Text-Driven Foley Sound Generation With Latent Diffusion Model](https://mlsanity.com/paper/2306.10359/) (2023)
- [AudioMorphix: Training-free audio editing with diffusion probabilistic models](https://mlsanity.com/paper/2505.16076/) (2025)
- [Bridging Paintings and Music -- Exploring Emotion based Music Generation through Paintings](https://mlsanity.com/paper/2409.07827/) (2024)
- [Latent Diffusion Model Based Foley Sound Generation System For DCASE Challenge 2023 Task 7](https://mlsanity.com/paper/2305.15905/) (2023)
- [DreamAudio: Customized Text-to-Audio Generation with Diffusion Models](https://mlsanity.com/paper/2509.06027/) (2025)
- [EmoHeal: An End-to-End System for Personalized Therapeutic Music Retrieval from Fine-grained Emotions](https://mlsanity.com/paper/2509.15986/) (2025)

## Abstract (arXiv)

Deep neural networks have recently achieved breakthroughs in sound generation. Despite the outstanding sample quality, current sound generation models face issues on small-scale datasets (e.g., overfitting), significantly limiting performance. In this paper, we make the first attempt to investigate the benefits of pre-training on sound generation with AudioLDM, the cutting-edge model for audio generation, as the backbone. Our study demonstrates the advantages of the pre-trained AudioLDM, especially in data-scarcity scenarios. In addition, the baselines and evaluation protocol for sound generation systems are not consistent enough to compare different studies directly. Aiming to facilitate further study on sound generation tasks, we benchmark the sound generation task on various frequently-used datasets. We hope our results on transfer learning and benchmarks can provide references for further research on conditional sound generation.