# Off-manifold robustness in synthesizer inversion with joint distribution flow matching

arXiv: https://arxiv.org/abs/2609.29320 · HTML: https://mlsanity.com/paper/2609.29320/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 21 of 21)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models](https://mlsanity.com/paper/2404.07724/) (2024)
- [One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale](https://mlsanity.com/paper/2303.06555/) (2023)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think](https://mlsanity.com/paper/2507.01467/) (2025)
- [OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows](https://mlsanity.com/paper/2412.01169/) (2024)
- [A Review of Differentiable Digital Signal Processing for Music & Speech Synthesis](https://mlsanity.com/paper/2308.15422/) (2023)
- [I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch](https://mlsanity.com/paper/2012.04572/) (2020)
- [Sinusoidal Frequency Estimation by Gradient Descent](https://mlsanity.com/paper/2210.14476/) (2022)
- [DDX7: Differentiable FM Synthesis of Musical Instrument Sounds](https://mlsanity.com/paper/2208.06169/) (2022)
- [InverSynth: Deep Estimation of Synthesizer Parameter Configurations from Audio Signals](https://mlsanity.com/paper/1812.06349/) (2018)
- [Sound2Synth: Interpreting Sound via FM Synthesizer Parameters Estimation](https://mlsanity.com/paper/2205.03043/) (2022)
- [Audio synthesizer inversion in symmetric parameter spaces with approximately equivariant flow matching](https://mlsanity.com/paper/2506.07199/) (2025)
- [Synthesizer Sound Matching Using Audio Spectrogram Transformers](https://mlsanity.com/paper/2407.16643/) (2024)
- [Synthesizer Preset Interpolation using Transformer Auto-Encoders](https://mlsanity.com/paper/2210.16984/) (2022)

## Abstract (arXiv)

Recent work on synthesizer inversion shows that generative models outperform deterministic approaches by explicitly modeling the ambiguity in mapping audio to parameters. Training such models, however, requires audio-parameter pairs, which are typically obtained by rendering sampled or preset parameters through the synthesizer itself. This creates a train-test mismatch that can degrade performance on off-manifold real-world recordings, for which ground-truth parameter annotations do not exist. To circumvent this obstacle, we propose to model the joint distribution of audio and parameters with a multi-modal continuous normalizing flow using independent noise schedules for each modality. This formulation allows us to train joint and conditional densities with paired synthesizer data, while unpaired real recordings can train the audio marginal alone, exposing the model to off-manifold signals without requiring parameter labels. Further, because the model learns to map from audio to parameters at all noise levels, we find that partially noising the audio reference at inference improves real-audio reconstruction, consistent with reducing sensitivity to distribution-specific detail while preserving coarse structure. Evaluating on Surge XT and Dexed, we find that modelling the joint distribution substantially improves both inversion of real-world and in-domain audio.