# Real-time Timbre Remapping with Differentiable DSP

arXiv: https://arxiv.org/abs/2407.04547 · HTML: https://mlsanity.com/paper/2407.04547/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 64 references that are in the map.
- First posted on arXiv 5 Jul 2024.

## Builds on (most cited first, 14 of 14)

- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks](https://mlsanity.com/paper/2008.12073/) (2020)
- [TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer](https://mlsanity.com/paper/1811.09620/) (2018)
- [Neural source-filter-based waveform model for statistical parametric speech synthesis](https://mlsanity.com/paper/1810.11946/) (2018)
- [A Review of Differentiable Digital Signal Processing for Music & Speech Synthesis](https://mlsanity.com/paper/2308.15422/) (2023)
- [Universal audio synthesizer control with normalizing flows](https://mlsanity.com/paper/1907.00971/) (2019)
- [I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch](https://mlsanity.com/paper/2012.04572/) (2020)
- [Music Style Transfer: A Position Paper](https://mlsanity.com/paper/1803.06841/) (2018)
- [Sinusoidal Frequency Estimation by Gradient Descent](https://mlsanity.com/paper/2210.14476/) (2022)
- [DDX7: Differentiable FM Synthesis of Musical Instrument Sounds](https://mlsanity.com/paper/2208.06169/) (2022)
- [Neural Waveshaping Synthesis](https://mlsanity.com/paper/2107.05050/) (2021)
- [NoiseBandNet: Controllable Time-Varying Neural Synthesis of Sound Effects Using Filterbanks](https://mlsanity.com/paper/2307.08007/) (2023)
- [Singing Voice Synthesis Using Differentiable LPC and Glottal-Flow-Inspired Wavetables](https://mlsanity.com/paper/2306.17252/) (2023)
- [Real-time Percussive Technique Recognition and Embedding Learning for the Acoustic Guitar](https://mlsanity.com/paper/2307.07426/) (2023)

## Built on by (most cited first, 3 of 3)

- [Designing Neural Synthesizers for Low-Latency Interaction](https://mlsanity.com/paper/2503.11562/) (2025)
- [SCRAPL: Scattering Transform with Random Paths for Machine Learning](https://mlsanity.com/paper/2602.11145/) (2026)
- [Evaluating Loss Functions in Differentiable Out-of-Domain Sound-Matching with Partial Parameter Distance](https://mlsanity.com/paper/2608.27698/) (2026)

## Abstract (arXiv)

Timbre is a primary mode of expression in diverse musical contexts. However, prevalent audio-driven synthesis methods predominantly rely on pitch and loudness envelopes, effectively flattening timbral expression from the input. Our approach draws on the concept of timbre analogies and investigates how timbral expression from an input signal can be mapped onto controls for a synthesizer. Leveraging differentiable digital signal processing, our method facilitates direct optimization of synthesizer parameters through a novel feature difference loss. This loss function, designed to learn relative timbral differences between musical events, prioritizes the subtleties of graded timbre modulations within phrases, allowing for meaningful translations in a timbre space. Using snare drum performances as a case study, where timbral expression is central, we demonstrate real-time timbre remapping from acoustic snare drums to a differentiable synthesizer modeled after the Roland TR-808.