# A Review of Differentiable Digital Signal Processing for Music & Speech Synthesis

arXiv: https://arxiv.org/abs/2308.15422 · HTML: https://mlsanity.com/paper/2308.15422/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2023.

## Built on by (most cited first, 20 of 20)

- [Unsupervised Harmonic Parameter Estimation Using Differentiable DSP and Spectral Optimal Transport](https://mlsanity.com/paper/2312.14507/) (2023)
- [Audio synthesizer inversion in symmetric parameter spaces with approximately equivariant flow matching](https://mlsanity.com/paper/2506.07199/) (2025)
- [TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrument](https://mlsanity.com/paper/2502.08939/) (2025)
- [Differentiable Modal Synthesis for Physical Modeling of Planar String Sound and Motion Simulation](https://mlsanity.com/paper/2407.05516/) (2024)
- [Real-time Timbre Remapping with Differentiable DSP](https://mlsanity.com/paper/2407.04547/) (2024)
- [Fast, High-Quality and Parameter-Efficient Articulatory Synthesis using Differentiable DSP](https://mlsanity.com/paper/2409.02451/) (2024)
- [The Inverse Drum Machine: Source Separation Through Joint Transcription and Analysis-by-Synthesis](https://mlsanity.com/paper/2505.03337/) (2025)
- [PESTO: Real-Time Pitch Estimation with Self-supervised Transposition-equivariant Objective](https://mlsanity.com/paper/2508.01488/) (2025)
- [DiTSinger: Scaling Singing Voice Synthesis with Diffusion Transformer and Implicit Alignment](https://mlsanity.com/paper/2510.09016/) (2025)
- [Generating Sample-Based Musical Instruments Using Neural Audio Codec Language Models](https://mlsanity.com/paper/2407.15641/) (2024)
- [Designing Neural Synthesizers for Low-Latency Interaction](https://mlsanity.com/paper/2503.11562/) (2025)
- [Modulation Discovery with Differentiable Digital Signal Processing](https://mlsanity.com/paper/2510.06204/) (2025)
- [Spectrotemporal Modulation: Efficient and Interpretable Feature Representation for Classifying Speech, Music, and Environmental Sounds](https://mlsanity.com/paper/2505.23509/) (2025)
- [Gradient-based Optimisation of Modulation Effects](https://mlsanity.com/paper/2601.04867/) (2026)
- [Lightweight Self-Supervised Detection of Fundamental Frequency and Accurate Probability of Voicing in Monophonic Music](https://mlsanity.com/paper/2601.11768/) (2026)
- [SCRAPL: Scattering Transform with Random Paths for Machine Learning](https://mlsanity.com/paper/2602.11145/) (2026)
- [Architecture and Affordances of PLAUD: Performative Latents and Unsupervised DDSP](https://mlsanity.com/paper/2608.13724/) (2026)
- [Differentiable Digital Signal Processing Mixture Model-Guided Diffusion for Synthesis Parameter Estimation from Harmonic Sound Mixtures](https://mlsanity.com/paper/2609.14427/) (2026)
- [Off-manifold robustness in synthesizer inversion with joint distribution flow matching](https://mlsanity.com/paper/2609.29320/) (2026)
- [GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding](https://mlsanity.com/paper/2610.04651/) (2026)

## Abstract (arXiv)

The term "differentiable digital signal processing" describes a family of techniques in which loss function gradients are backpropagated through digital signal processors, facilitating their integration into neural networks. This article surveys the literature on differentiable audio signal processing, focusing on its use in music & speech synthesis. We catalogue applications to tasks including music performance rendering, sound matching, and voice transformation, discussing the motivations for and implications of the use of this methodology. This is accompanied by an overview of digital signal processing operations that have been implemented differentiably. Finally, we highlight open challenges, including optimisation pathologies, robustness to real-world conditions, and design trade-offs, and discuss directions for future research.