# Differentiable Modal Synthesis for Physical Modeling of Planar String Sound and Motion Simulation

arXiv: https://arxiv.org/abs/2407.05516 · HTML: https://mlsanity.com/paper/2407.05516/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 66 references that are in the map.
- First posted on arXiv 7 Jul 2024.

## Builds on (most cited first, 23 of 23)

- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [GANSynth: Adversarial Neural Audio Synthesis](https://mlsanity.com/paper/1902.08710/) (2019)
- [CREPE: A Convolutional Representation for Pitch Estimation](https://mlsanity.com/paper/1802.06182/) (2018)
- [LPCNet: Improving Neural Speech Synthesis Through Linear Prediction](https://mlsanity.com/paper/1810.11846/) (2018)
- [Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations](https://mlsanity.com/paper/2110.14513/) (2021)
- [Neural source-filter waveform models for statistical parametric speech synthesis](https://mlsanity.com/paper/1904.12088/) (2019)
- [NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis](https://mlsanity.com/paper/2211.09407/) (2022)
- [MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling](https://mlsanity.com/paper/2112.09312/) (2021)
- [A Review of Differentiable Digital Signal Processing for Music & Speech Synthesis](https://mlsanity.com/paper/2308.15422/) (2023)
- [I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch](https://mlsanity.com/paper/2012.04572/) (2020)
- [Sinusoidal Frequency Estimation by Gradient Descent](https://mlsanity.com/paper/2210.14476/) (2022)
- [DDX7: Differentiable FM Synthesis of Musical Instrument Sounds](https://mlsanity.com/paper/2208.06169/) (2022)
- [Neural Waveshaping Synthesis](https://mlsanity.com/paper/2107.05050/) (2021)
- [Differentiable Artificial Reverberation](https://mlsanity.com/paper/2105.13940/) (2021)
- [Physical Modeling using Recurrent Neural Networks with Fast Convolutional Layers](https://mlsanity.com/paper/2204.10125/) (2022)
- [Unsupervised Harmonic Parameter Estimation Using Differentiable DSP and Spectral Optimal Transport](https://mlsanity.com/paper/2312.14507/) (2023)
- [Rigid-Body Sound Synthesis with Differentiable Modal Resonators](https://mlsanity.com/paper/2210.15306/) (2022)
- [NoiseBandNet: Controllable Time-Varying Neural Synthesis of Sound Effects Using Filterbanks](https://mlsanity.com/paper/2307.08007/) (2023)
- [String Sound Synthesizer on GPU-accelerated Finite Difference Scheme](https://mlsanity.com/paper/2311.18505/) (2023)

## Built on by (most cited first, 4 of 4)

- [Fast Differentiable Modal Simulation of Non-linear Strings, Membranes, and Plates](https://mlsanity.com/paper/2505.05940/) (2025)
- [Learning Nonlinear Dynamics in Physical Modelling Synthesis using Neural Ordinary Differential Equations](https://mlsanity.com/paper/2505.10511/) (2025)
- [Stable Differentiable Modal Synthesis for Learning Nonlinear Dynamics](https://mlsanity.com/paper/2601.10453/) (2026)
- [Identifying the nonlinear string dynamics with port-Hamiltonian neural networks](https://mlsanity.com/paper/2605.12785/) (2026)

## Abstract (arXiv)

While significant advancements have been made in music generation and differentiable sound synthesis within machine learning and computer audition, the simulation of instrument vibration guided by physical laws has been underexplored. To address this gap, we introduce a novel model for simulating the spatio-temporal motion of nonlinear strings, integrating modal synthesis and spectral modeling within a neural network framework. Our model leverages physical properties and fundamental frequencies as inputs, outputting string states across time and space that solve the partial differential equation characterizing the nonlinear string. Empirical evaluations demonstrate that the proposed architecture achieves superior accuracy in string motion simulation compared to existing baseline architectures. The code and demo are available online.