# DDX7: Differentiable FM Synthesis of Musical Instrument Sounds

arXiv: https://arxiv.org/abs/2208.06169 · HTML: https://mlsanity.com/paper/2208.06169/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 56 references that are in the map.
- First posted on arXiv 12 Aug 2022.

## Builds on (most cited first, 22 of 22)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [GANSynth: Adversarial Neural Audio Synthesis](https://mlsanity.com/paper/1902.08710/) (2019)
- [SampleRNN: An Unconditional End-to-End Neural Audio Generation Model](https://mlsanity.com/paper/1612.07837/) (2016)
- [CREPE: A Convolutional Representation for Pitch Estimation](https://mlsanity.com/paper/1802.06182/) (2018)
- [RAVE: A variational autoencoder for fast and high-quality neural audio synthesis](https://mlsanity.com/paper/2111.05011/) (2021)
- [Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications](https://mlsanity.com/paper/1612.08727/) (2016)
- [Neural source-filter waveform models for statistical parametric speech synthesis](https://mlsanity.com/paper/1904.12088/) (2019)
- [DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks](https://mlsanity.com/paper/2008.12073/) (2020)
- [TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer](https://mlsanity.com/paper/1811.09620/) (2018)
- [MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling](https://mlsanity.com/paper/2112.09312/) (2021)
- [Universal audio synthesizer control with normalizing flows](https://mlsanity.com/paper/1907.00971/) (2019)
- [I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch](https://mlsanity.com/paper/2012.04572/) (2020)
- [Efficient neural networks for real-time modeling of analog dynamic range compression](https://mlsanity.com/paper/2102.06200/) (2021)
- [Self-Supervised VQ-VAE for One-Shot Music Style Transfer](https://mlsanity.com/paper/2102.05749/) (2021)
- [Neural Waveshaping Synthesis](https://mlsanity.com/paper/2107.05050/) (2021)
- [Hierarchical Timbre-Painting and Articulation Generation](https://mlsanity.com/paper/2008.13095/) (2020)
- [Sound2Synth: Interpreting Sound via FM Synthesizer Parameters Estimation](https://mlsanity.com/paper/2205.03043/) (2022)
- [StyleWaveGAN: Style-based synthesis of drum sounds with extensive controls using generative adversarial networks](https://mlsanity.com/paper/2204.00907/) (2022)

## Built on by (most cited first, 13 of 13)

- [DiffMoog: a Differentiable Modular Synthesizer for Sound Matching](https://mlsanity.com/paper/2401.12570/) (2024)
- [TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrument](https://mlsanity.com/paper/2502.08939/) (2025)
- [Differentiable Modal Synthesis for Physical Modeling of Planar String Sound and Motion Simulation](https://mlsanity.com/paper/2407.05516/) (2024)
- [Synthesizer Sound Matching Using Audio Spectrogram Transformers](https://mlsanity.com/paper/2407.16643/) (2024)
- [Synthesizer Preset Interpolation using Transformer Auto-Encoders](https://mlsanity.com/paper/2210.16984/) (2022)
- [Real-time Percussive Technique Recognition and Embedding Learning for the Acoustic Guitar](https://mlsanity.com/paper/2307.07426/) (2023)
- [Real-time Timbre Remapping with Differentiable DSP](https://mlsanity.com/paper/2407.04547/) (2024)
- [Sines, Transient, Noise Neural Modeling of Piano Notes](https://mlsanity.com/paper/2409.06513/) (2024)
- [Modulation Discovery with Differentiable Digital Signal Processing](https://mlsanity.com/paper/2510.06204/) (2025)
- [GuitarFlow: Realistic Electric Guitar Synthesis From Tablatures via Flow Matching and Style Transfer](https://mlsanity.com/paper/2510.21872/) (2025)
- [NAS-FM: Neural Architecture Search for Tunable and Interpretable Sound Synthesis based on Frequency Modulation](https://mlsanity.com/paper/2305.12868/) (2023)
- [Neural Proxies for Sound Synthesizers: Learning Perceptually Informed Preset Representations](https://mlsanity.com/paper/2509.07635/) (2025)
- [Off-manifold robustness in synthesizer inversion with joint distribution flow matching](https://mlsanity.com/paper/2609.29320/) (2026)

## Abstract (arXiv)

FM Synthesis is a well-known algorithm used to generate complex timbre from a compact set of design primitives. Typically featuring a MIDI interface, it is usually impractical to control it from an audio source. On the other hand, Differentiable Digital Signal Processing (DDSP) has enabled nuanced audio rendering by Deep Neural Networks (DNNs) that learn to control differentiable synthesis layers from arbitrary sound inputs. The training process involves a corpus of audio for supervision, and spectral reconstruction loss functions. Such functions, while being great to match spectral amplitudes, present a lack of pitch direction which can hinder the joint optimization of the parameters of FM synthesizers. In this paper, we take steps towards enabling continuous control of a well-established FM synthesis architecture from an audio input. Firstly, we discuss a set of design constraints that ease spectral optimization of a differentiable FM synthesizer via a standard reconstruction loss. Next, we present Differentiable DX7 (DDX7), a lightweight architecture for neural FM resynthesis of musical instrument sounds in terms of a compact set of parameters. We train the model on instrument samples extracted from the URMP dataset, and quantitatively demonstrate its comparable audio quality against selected benchmarks.