# Improving Inference-Time Optimisation for Vocal Effects Style Transfer with a Gaussian Prior

arXiv: https://arxiv.org/abs/2505.11315 · HTML: https://mlsanity.com/paper/2505.11315/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 16 May 2025.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [A Variational Perspective on Solving Inverse Problems with Diffusion Models](https://mlsanity.com/paper/2305.04391/) (2023)
- [DITTO: Diffusion Inference-Time T-Optimization for Music Generation](https://mlsanity.com/paper/2401.12179/) (2024)
- [Multi-Source Diffusion Models for Simultaneous Music Generation and Separation](https://mlsanity.com/paper/2302.02257/) (2023)
- [Diffusion Models for Audio Restoration](https://mlsanity.com/paper/2402.09821/) (2024)
- [Balancing reconstruction error and Kullback-Leibler divergence in Variational Autoencoders](https://mlsanity.com/paper/2002.07514/) (2020)
- [Automatic multitrack mixing with a differentiable mixing console of neural audio effects](https://mlsanity.com/paper/2010.10291/) (2020)
- [DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generation](https://mlsanity.com/paper/2405.20289/) (2024)
- [Neural source-filter-based waveform model for statistical parametric speech synthesis](https://mlsanity.com/paper/1810.11946/) (2018)
- [Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolution](https://mlsanity.com/paper/2210.15793/) (2022)
- [Style Transfer of Audio Effects with Differentiable Signal Processing](https://mlsanity.com/paper/2207.08759/) (2022)
- [Music Mixing Style Transfer: A Contrastive Learning Approach to Disentangle Audio Effects](https://mlsanity.com/paper/2211.02247/) (2022)
- [Automatic music mixing with deep learning and out-of-domain data](https://mlsanity.com/paper/2208.11428/) (2022)
- [VRDMG: Vocal Restoration via Diffusion Posterior Sampling with Multiple Guidance](https://mlsanity.com/paper/2309.06934/) (2023)
- [Text2FX: Harnessing CLAP Embeddings for Text-Guided Audio Effects](https://mlsanity.com/paper/2409.18847/) (2024)
- [ST-ITO: Controlling Audio Effects for Style Transfer with Inference-Time Optimization](https://mlsanity.com/paper/2410.21233/) (2024)
- [Diff-MST: Differentiable Mixing Style Transfer](https://mlsanity.com/paper/2407.08889/) (2024)
- [Blind Estimation of Audio Processing Graph](https://mlsanity.com/paper/2303.08610/) (2023)
- [Blind estimation of audio effects using an auto-encoder approach and differentiable digital signal processing](https://mlsanity.com/paper/2310.11781/) (2023)
- [Learning to Solve Inverse Problems for Perceptual Sound Matching](https://mlsanity.com/paper/2311.14213/) (2023)
- [The Responsibility Problem in Neural Networks with Unordered Targets](https://mlsanity.com/paper/2304.09499/) (2023)
- [DiffVox: A Differentiable Model for Capturing and Analysing Vocal Effects Distributions](https://mlsanity.com/paper/2504.14735/) (2025)
- [The Role of Communication and Reference Songs in the Mixing Process: Insights from Professional Mix Engineers](https://mlsanity.com/paper/2309.03404/) (2023)

## Abstract (arXiv)

Style Transfer with Inference-Time Optimisation (ST-ITO) is a recent approach for transferring the applied effects of a reference audio to an audio track. It optimises the effect parameters to minimise the distance between the style embeddings of the processed audio and the reference. However, this method treats all possible configurations equally and relies solely on the embedding space, which can result in unrealistic configurations or biased outcomes. We address this pitfall by introducing a Gaussian prior derived from the DiffVox vocal preset dataset over the parameter space. The resulting optimisation is equivalent to maximum-a-posteriori estimation. Evaluations on vocal effects transfer on the MedleyDB dataset show significant improvements across metrics compared to baselines, including a blind audio effects estimator, nearest-neighbour approaches, and uncalibrated ST-ITO. The proposed calibration reduces the parameter mean squared error by up to 33% and more closely matches the reference style. Subjective evaluations with 16 participants confirm the superiority of our method in limited data regimes. This work demonstrates how incorporating prior knowledge at inference time enhances audio effects transfer, paving the way for more effective and realistic audio processing systems.