# Exploring Efficient Waveform Diffusion Models for Foley Sound Generation

arXiv: https://arxiv.org/abs/2607.29148 · HTML: https://mlsanity.com/paper/2607.29148/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 21 of its 24 references that are in the map.
- First posted on arXiv 31 Jul 2026.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 21 of 21)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms](https://mlsanity.com/paper/1812.08466/) (2018)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [Solving Audio Inverse Problems with a Diffusion Model](https://mlsanity.com/paper/2210.15228/) (2022)
- [A Study on Speech Enhancement Based on Diffusion Probabilistic Model](https://mlsanity.com/paper/2107.11876/) (2021)
- [Full-band General Audio Synthesis with Score-based Diffusion](https://mlsanity.com/paper/2210.14661/) (2022)
- [General-purpose Tagging of Freesound Audio with AudioSet Labels: Task Description, Dataset, and Baseline](https://mlsanity.com/paper/1807.09902/) (2018)
- [FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos](https://mlsanity.com/paper/2107.09262/) (2021)
- [Foley Sound Synthesis at the DCASE 2023 Challenge](https://mlsanity.com/paper/2304.12521/) (2023)
- [T-FOLEY: A Controllable Waveform-Domain Diffusion Model for Temporal-Event-Guided Foley Sound Synthesis](https://mlsanity.com/paper/2401.09294/) (2024)
- [Generative Speech Foundation Model Pretraining for High-Quality Speech Extraction and Restoration](https://mlsanity.com/paper/2409.16117/) (2024)
- [FreGrad: Lightweight and Fast Frequency-aware Diffusion Vocoder](https://mlsanity.com/paper/2401.10032/) (2024)
- [Text-Driven Foley Sound Generation With Latent Diffusion Model](https://mlsanity.com/paper/2306.10359/) (2023)
- [UnDiff: Unsupervised Voice Restoration with Unconditional Diffusion Model](https://mlsanity.com/paper/2306.00721/) (2023)
- [Rhythmic Foley: A Framework For Seamless Audio-Visual Alignment In Video-to-Audio Synthesis](https://mlsanity.com/paper/2409.08628/) (2024)
- [MambaFoley: Foley Sound Generation using Selective State-Space Models](https://mlsanity.com/paper/2409.09162/) (2024)
- [Unsupervised Single-Channel Audio Separation with Diffusion Source Priors](https://mlsanity.com/paper/2512.07226/) (2025)

## Abstract (arXiv)

Recent advances in diffusion models have enabled high-fidelity Foley sound generation directly in the waveform space. Existing waveform diffusion models primarily rely on time-domain architectures, such as CNN-based U-Nets and DiffWave-style models, or frequency-domain Transformers modeling temporal dependencies. However, these systems are typically built with large model capacities and substantial computational costs, leaving compact and efficient waveform diffusion architectures largely underexplored. In this work, we introduce a Dual-Path (DP) architecture for waveform diffusion that performs dimension-wise self-attention along both subband and frame axes in the time-frequency domain. This DP design enables fine-grained temporal-spectral modeling while maintaining high efficiency. Based on the proposed DP backbone, we develop two variants: DP-DiT and DP-U-Net. Experiments on the DCASE and FSD-Kaggle2018 datasets demonstrate their superior performance. Notably, the 3M parameter variant achieves performance comparable to models with more than 50M parameters. Audio samples are available at https://samplesdemo.github.io/DP-Foley/.