# Speech Synthesis and Control Using Differentiable DSP

arXiv: https://arxiv.org/abs/2010.15084 · HTML: https://mlsanity.com/paper/2010.15084/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2020.

## Built on by (most cited first, 8 of 8)

- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [InstrumentGen: Generating Sample-Based Musical Instruments From Text](https://mlsanity.com/paper/2311.04339/) (2023)
- [Latent Space Explorations of Singing Voice Synthesis using DDSP](https://mlsanity.com/paper/2103.07197/) (2021)
- [Differentiable WORLD Synthesizer-based Neural Vocoder With Application To End-To-End Audio Style Transfer](https://mlsanity.com/paper/2208.07282/) (2022)
- [Fast, High-Quality and Parameter-Efficient Articulatory Synthesis using Differentiable DSP](https://mlsanity.com/paper/2409.02451/) (2024)
- [PESTO: Real-Time Pitch Estimation with Self-supervised Transposition-equivariant Objective](https://mlsanity.com/paper/2508.01488/) (2025)
- [Embedding a Differentiable Mel-cepstral Synthesis Filter to a Neural Speech Synthesis System](https://mlsanity.com/paper/2211.11222/) (2022)
- [A General Framework for Learning Procedural Audio Models of Environmental Sounds](https://mlsanity.com/paper/2303.02396/) (2023)

## Abstract (arXiv)

Modern text-to-speech systems are able to produce natural and high-quality speech, but speech contains factors of variation (e.g. pitch, rhythm, loudness, timbre) that text alone cannot contain. In this work we move towards a speech synthesis system that can produce diverse speech renditions of a text by allowing (but not requiring) explicit control over the various factors of variation. We propose a new neural vocoder that offers control of such factors of variation. This is achieved by employing differentiable digital signal processing (DDSP) (previously used only for music rather than speech), which exposes these factors of variation. The results show that the proposed approach can produce natural speech with realistic timbre, and individual factors of variation can be freely controlled.