# Music Restoration via Latent Operator Optimization and Diffusion Model Priors

arXiv: https://arxiv.org/abs/2608.01972 · HTML: https://mlsanity.com/paper/2608.01972/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 43 of its 49 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 43)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [Stable Audio Open](https://mlsanity.com/paper/2407.14358/) (2024)
- [Fast Timing-Conditioned Latent Audio Diffusion](https://mlsanity.com/paper/2402.04825/) (2024)
- [A Survey on Diffusion Models for Inverse Problems](https://mlsanity.com/paper/2410.00083/) (2024)
- [Adapting Frechet Audio Distance for Generative Music Evaluation](https://mlsanity.com/paper/2311.01616/) (2023)
- [AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models](https://mlsanity.com/paper/2304.00830/) (2023)
- [Improving the Diffusability of Autoencoders](https://mlsanity.com/paper/2502.14831/) (2025)
- [Solving Audio Inverse Problems with a Diffusion Model](https://mlsanity.com/paper/2210.15228/) (2022)
- [Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus](https://mlsanity.com/paper/2112.10358/) (2021)
- [VoiceFixer: Toward General Speech Restoration with Neural Vocoder](https://mlsanity.com/paper/2109.13731/) (2021)
- [Fast Diffusion EM: a diffusion model for blind inverse problems with application to deconvolution](https://mlsanity.com/paper/2309.00287/) (2023)
- [Music2Latent: Consistency Autoencoders for Latent Audio Compression](https://mlsanity.com/paper/2408.06500/) (2024)
- [KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generation](https://mlsanity.com/paper/2502.15602/) (2025)
- [SILO: Solving Inverse Problems with Latent Operators](https://mlsanity.com/paper/2501.11746/) (2025)
- [NHSS: A Speech and Singing Parallel Database](https://mlsanity.com/paper/2012.00337/) (2020)
- [A Diffusion-Based Generative Equalizer for Music Restoration](https://mlsanity.com/paper/2403.18636/) (2024)
- [Automatic music mixing with deep learning and out-of-domain data](https://mlsanity.com/paper/2208.11428/) (2022)
- [towards automatic transcription of polyphonic electric guitar music:a new dataset and a multi-loss transformer model](https://mlsanity.com/paper/2202.09907/) (2022)
- [Text2FX: Harnessing CLAP Embeddings for Text-Guided Audio Effects](https://mlsanity.com/paper/2409.18847/) (2024)
- [Apollo: Band-sequence Modeling for High-Quality Audio Restoration](https://mlsanity.com/paper/2409.08514/) (2024)
- [High-Resolution Speech Restoration with Latent Diffusion Model](https://mlsanity.com/paper/2409.11145/) (2024)
- [General Purpose Audio Effect Removal](https://mlsanity.com/paper/2308.16177/) (2023)
- [ST-ITO: Controlling Audio Effects for Style Transfer with Inference-Time Optimization](https://mlsanity.com/paper/2410.21233/) (2024)
- [ArrayDPS: Unsupervised Blind Speech Separation with a Diffusion Prior](https://mlsanity.com/paper/2505.05657/) (2025)

## Abstract (arXiv)

Music restoration seeks to recover a clean signal from an observed recording degraded by an unknown effect, distortion, or corruption. Existing systems often rely on paired training data and distortion-specific supervision, which limits their use when the forward process is not known in advance. We propose LOUDAR (Latent-space Optimization of Unknown Distortion for Audio Restoration) a general-purpose restoration method that operates in the latent space of a pretrained audio autoencoder and models the unknown distortion as a learnable latent operator. At inference time, LOUDAR alternates between estimating the clean latent variable and updating the latent operator parameters. An unconditional latent diffusion model provides a prior over clean audio and regularizes this inference by steering the latent estimate toward the manifold of clean recordings. Because the degradation model is adapted per input, the approach is broadly applicable across diverse restoration problems. We evaluate LOUDAR on singing voice effect removal and restoration, as well as guitar distortion removal, and show that it consistently improves over degraded inputs and is competitive with supervised and unsupervised baselines in waveform and latent domains.