# Reconstructing the Vocal Tract with Differentiable Acoustic Simulation

arXiv: https://arxiv.org/abs/2609.36737 · HTML: https://mlsanity.com/paper/2609.36737/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 59 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 30 of 31)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines](https://mlsanity.com/paper/2010.11567/) (2020)
- [LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus](https://mlsanity.com/paper/2305.18802/) (2023)
- [Learning Neural Acoustic Fields](https://mlsanity.com/paper/2204.00628/) (2022)
- [TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudio](https://mlsanity.com/paper/2304.01448/) (2023)
- [AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis](https://mlsanity.com/paper/2302.02088/) (2023)
- [Neural source-filter waveform models for statistical parametric speech synthesis](https://mlsanity.com/paper/1904.12088/) (2019)
- [JVS corpus: free Japanese multi-speaker voice corpus](https://mlsanity.com/paper/1908.06248/) (2019)
- [A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric images](https://mlsanity.com/paper/2102.07896/) (2021)
- [Coding Speech through Vocal Tract Kinematics](https://mlsanity.com/paper/2406.12998/) (2024)
- [Acoustic Volume Rendering for Neural Impulse Response Fields](https://mlsanity.com/paper/2411.06307/) (2024)
- [Hearing Anything Anywhere](https://mlsanity.com/paper/2406.07532/) (2024)
- [Phonetic Segmentation of the UCLA Phonetics Lab Archive](https://mlsanity.com/paper/2403.19509/) (2024)
- [Unsupervised Music Source Separation Using Differentiable Parametric Source Models](https://mlsanity.com/paper/2201.09592/) (2022)
- [Speech2rtMRI: Speech-Guided Diffusion Model for Real-time MRI Video of the Vocal Tract during Speech](https://mlsanity.com/paper/2409.15525/) (2024)
- [Articulation GAN: Unsupervised modeling of articulatory learning](https://mlsanity.com/paper/2210.15173/) (2022)
- [SwiftF0: Fast and Accurate Monophonic Pitch Detection](https://mlsanity.com/paper/2508.18440/) (2025)
- [Vocal Tract Area Estimation by Gradient Descent](https://mlsanity.com/paper/2307.04702/) (2023)

## Abstract (arXiv)

The vocal tract is the region of the human body responsible for filtering one's voice to create speech. In this paper, we present a differentiable and GPU accelerated acoustic simulator for the vocal tract. The differentiable simulator synthesizes speech by propagating sound along an acoustic tube model of the vocal tract, and via its gradients, can solve the inverse problem: reconstructing the shape of the vocal tract solely from the sound it produces. Although the inverse mapping between geometry and sound is notoriously non-convex, we discover that gradient descent succeeds with three technical contributions: (1) we design a frequency domain formulation of the vocal tract's fluid dynamics that is 70x more GPU parallelizable than finite differences in time, (2) we integrate a differentiable model for turbulence to synthesize consonants, and (3) similar to prior work in implicit neural representations (INRs) and neural fields, we find that parameterizing the geometry with a neural network accelerates convergence and escapes local minima that trap discrete representations. Because the simulator is differentiable, it is readily integrated with other deep learning pipelines to enable novel linguistics and medical imaging applications. (1) We demonstrate self-supervised autoencoding of vocal tract shapes across 11 languages, and (2) we couple our simulator with a generative model of MRI (magnetic resonance imaging) images to reconstruct one's moving vocal tract from only their speech without paired data.