# DDSynth-RL: Audio Synthesizer Inversion via Discrete Diffusion with Reinforcement Learning

arXiv: https://arxiv.org/abs/2608.03032 · HTML: https://mlsanity.com/paper/2608.03032/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 24 of 24)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models](https://mlsanity.com/paper/2211.15029/) (2022)
- [d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning](https://mlsanity.com/paper/2504.12216/) (2025)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Protein Design with Guided Discrete Diffusion](https://mlsanity.com/paper/2305.20009/) (2023)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [ChatMusician: Understanding and Generating Music Intrinsically with LLM](https://mlsanity.com/paper/2402.16153/) (2024)
- [DPLM-2: A Multimodal Diffusion Protein Language Model](https://mlsanity.com/paper/2410.13782/) (2024)
- [MuseCoco: Generating Symbolic Music from Text](https://mlsanity.com/paper/2306.00110/) (2023)
- [MusicRL: Aligning Music Generation to Human Preferences](https://mlsanity.com/paper/2402.04229/) (2024)
- [NotaGen: Advancing Musicality in Symbolic Music Generation with Large Language Model Training Paradigms](https://mlsanity.com/paper/2502.18008/) (2025)
- [Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policies](https://mlsanity.com/paper/2510.05725/) (2025)
- [A Unified Model for Zero-shot Music Source Separation, Transcription and Synthesis](https://mlsanity.com/paper/2108.03456/) (2021)
- [Discrete Diffusion Probabilistic Models for Symbolic Music Generation](https://mlsanity.com/paper/2305.09489/) (2023)
- [DawDreamer: Bridging the Gap Between Digital Audio Workstations and Python Interfaces](https://mlsanity.com/paper/2111.09931/) (2021)
- [Sound2Synth: Interpreting Sound via FM Synthesizer Parameters Estimation](https://mlsanity.com/paper/2205.03043/) (2022)
- [DiffMoog: a Differentiable Modular Synthesizer for Sound Matching](https://mlsanity.com/paper/2401.12570/) (2024)
- [Audio synthesizer inversion in symmetric parameter spaces with approximately equivariant flow matching](https://mlsanity.com/paper/2506.07199/) (2025)
- [Learning to Solve Inverse Problems for Perceptual Sound Matching](https://mlsanity.com/paper/2311.14213/) (2023)
- [Assessing the Alignment of Audio Representations with Timbre Similarity Ratings](https://mlsanity.com/paper/2507.07764/) (2025)
- [ViTex: Visual Texture Control for Multi-Track Symbolic Music Generation via Discrete Diffusion Models](https://mlsanity.com/paper/2603.01984/) (2026)

## Abstract (arXiv)

Synthesizer inversion is challenging for two main reasons: 1) Distinct parameter configurations can produce perceptually similar sounds. 2) Parameter-space losses often fail to reflect rendered audio similarity, while the synthesizer being a non-differentiable black box prevents simple audio-domain supervision. To address the one-to-many mapping induced by the first challenge, we formulate synthesizer inversion as conditional generation over discrete synthesizer parameters and use masked discrete diffusion as the generator. This treatment additionally avoids the fixed-order assumption of autoregressive models and the continuous-relaxation mismatch of flow matching when modeling categorical synthesizer controls. To address the second challenge, we further fine-tune the model with GRPO-style audio-domain rewards computed from rendered outputs. Experiments on Dexed show that, after supervised training, the discrete diffusion model is competitive with autoregressive and flow-matching baselines, and reward-based fine-tuning further improves out-of-domain audio matching performance. Code and demos are available at: https://github.com/DDSynth-RL/DDSynthRL.