# DDSP: Differentiable Digital Signal Processing

arXiv: https://arxiv.org/abs/2001.04643 · HTML: https://mlsanity.com/paper/2001.04643/

## Key facts

- Cited by 178 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jan 2020.

## Built on by (most cited first, 30 of 178)

- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [It's Raw! Audio Generation with State-Space Models](https://mlsanity.com/paper/2202.09729/) (2022)
- [Symbolic Music Generation with Diffusion Models](https://mlsanity.com/paper/2103.16091/) (2021)
- [Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens](https://mlsanity.com/paper/2503.01710/) (2025)
- [Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations](https://mlsanity.com/paper/2110.14513/) (2021)
- [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022)
- [RAVE: A variational autoencoder for fast and high-quality neural audio synthesis](https://mlsanity.com/paper/2111.05011/) (2021)
- [SingSong: Generating musical accompaniments from singing](https://mlsanity.com/paper/2301.12662/) (2023)
- [A Survey of Sound Source Localization with Deep Learning Methods](https://mlsanity.com/paper/2109.03465/) (2021)
- [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://mlsanity.com/paper/2206.05408/) (2022)
- [Relative Positional Encoding for Transformers with Linear Complexity](https://mlsanity.com/paper/2105.08399/) (2021)
- [MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling](https://mlsanity.com/paper/2112.09312/) (2021)
- [MusicRL: Aligning Music Generation to Human Preferences](https://mlsanity.com/paper/2402.04229/) (2024)
- [WavJourney: Compositional Audio Creation with Large Language Models](https://mlsanity.com/paper/2307.14335/) (2023)
- [Musika! Fast Infinite Waveform Music Generation](https://mlsanity.com/paper/2208.08706/) (2022)
- [Consistency Models for Scalable and Fast Simulation-Based Inference](https://mlsanity.com/paper/2312.05440/) (2023)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [Music Mixing Style Transfer: A Contrastive Learning Approach to Disentangle Audio Effects](https://mlsanity.com/paper/2211.02247/) (2022)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [Real-time Denoising and Dereverberation with Tiny Recurrent U-Net](https://mlsanity.com/paper/2102.03207/) (2021)
- [Evaluating generative audio systems and their metrics](https://mlsanity.com/paper/2209.00130/) (2022)
- [Bass Accompaniment Generation via Latent Diffusion](https://mlsanity.com/paper/2402.01412/) (2024)
- [Catch-A-Waveform: Learning to Generate Audio from a Single Short Example](https://mlsanity.com/paper/2106.06426/) (2021)
- [Self-Supervised VQ-VAE for One-Shot Music Style Transfer](https://mlsanity.com/paper/2102.05749/) (2021)

## Abstract (arXiv)

Most generative models of audio directly generate samples in one of two domains: time or frequency. While sufficient to express any signal, these representations are inefficient, as they do not utilize existing knowledge of how sound is generated and perceived. A third approach (vocoders/synthesizers) successfully incorporates strong domain knowledge of signal processing and perception, but has been less actively researched due to limited expressivity and difficulty integrating with modern auto-differentiation-based machine learning methods. In this paper, we introduce the Differentiable Digital Signal Processing (DDSP) library, which enables direct integration of classic signal processing elements with deep learning methods. Focusing on audio synthesis, we achieve high-fidelity generation without the need for large autoregressive models or adversarial losses, demonstrating that DDSP enables utilizing strong inductive biases without losing the expressive power of neural networks. Further, we show that combining interpretable modules permits manipulation of each separate model component, with applications such as independent control of pitch and loudness, realistic extrapolation to pitches not seen during training, blind dereverberation of room acoustics, transfer of extracted room acoustics to new environments, and transformation of timbre between disparate sources. In short, DDSP enables an interpretable and modular approach to generative modeling, without sacrificing the benefits of deep learning. The library is publicly available at https://github.com/magenta/ddsp and we welcome further contributions from the community and domain experts.