# Timbre transfer using image-to-image denoising diffusion implicit models

arXiv: https://arxiv.org/abs/2307.04586 · HTML: https://mlsanity.com/paper/2307.04586/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 10 Jul 2023.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 26 of 26)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [A Continuous Time Framework for Discrete Denoising Models](https://mlsanity.com/paper/2205.14987/) (2022)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Diffsound: Discrete Diffusion Model for Text-to-sound Generation](https://mlsanity.com/paper/2207.09983/) (2022)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Symbolic Music Generation with Diffusion Models](https://mlsanity.com/paper/2103.16091/) (2021)
- [Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2304.03322/) (2023)
- [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://mlsanity.com/paper/2206.05408/) (2022)
- [CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis](https://mlsanity.com/paper/2106.07431/) (2021)
- [TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer](https://mlsanity.com/paper/1811.09620/) (2018)
- [A Universal Music Translation Network](https://mlsanity.com/paper/1805.07848/) (2018)
- [Learning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoders](https://mlsanity.com/paper/1906.08152/) (2019)
- [Self-Supervised VQ-VAE for One-Shot Music Style Transfer](https://mlsanity.com/paper/2102.05749/) (2021)
- [Hierarchical Timbre-Painting and Articulation Generation](https://mlsanity.com/paper/2008.13095/) (2020)
- [Conditioning Autoencoder Latent Spaces for Real-Time Timbre Interpolation and Synthesis](https://mlsanity.com/paper/2001.11296/) (2020)

## Built on by (most cited first, 7 of 7)

- [Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfer](https://mlsanity.com/paper/2409.06096/) (2024)
- [SoundMorpher: Perceptually-Uniform Sound Morphing with Diffusion Model](https://mlsanity.com/paper/2410.02144/) (2024)
- [Diffusion Timbre Transfer Via Mutual Information Guided Inpainting](https://mlsanity.com/paper/2601.01294/) (2026)
- [WaveTransfer: A Flexible End-to-end Multi-instrument Timbre Transfer with Diffusion](https://mlsanity.com/paper/2409.15321/) (2024)
- [Towards HRTF Personalization using Denoising Diffusion Models](https://mlsanity.com/paper/2501.02871/) (2025)
- [Designing Neural Synthesizers for Low-Latency Interaction](https://mlsanity.com/paper/2503.11562/) (2025)
- [Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems](https://mlsanity.com/paper/2605.09259/) (2026)

## Abstract (arXiv)

Timbre transfer techniques aim at converting the sound of a musical piece generated by one instrument into the same one as if it was played by another instrument, while maintaining as much as possible the content in terms of musical characteristics such as melody and dynamics. Following their recent breakthroughs in deep learning-based generation, we apply Denoising Diffusion Models (DDMs) to perform timbre transfer. Specifically, we apply the recently proposed Denoising Diffusion Implicit Models (DDIMs) that enable to accelerate the sampling procedure. Inspired by the recent application of DDMs to image translation problems we formulate the timbre transfer task similarly, by first converting the audio tracks into log mel spectrograms and by conditioning the generation of the desired timbre spectrogram through the input timbre spectrogram. We perform both one-to-one and many-to-many timbre transfer, by converting audio waveforms containing only single instruments and multiple instruments, respectively. We compare the proposed technique with existing state-of-the-art methods both through listening tests and objective measures in order to demonstrate the effectiveness of the proposed model.