# TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer

arXiv: https://arxiv.org/abs/1811.09620 · HTML: https://mlsanity.com/paper/1811.09620/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2018.

## Built on by (most cited first, 30 of 33)

- [Solving Audio Inverse Problems with a Diffusion Model](https://mlsanity.com/paper/2210.15228/) (2022)
- [Conditional Generation of Audio from Video via Foley Analogies](https://mlsanity.com/paper/2304.08490/) (2023)
- [Self-Supervised VQ-VAE for One-Shot Music Style Transfer](https://mlsanity.com/paper/2102.05749/) (2021)
- [DDX7: Differentiable FM Synthesis of Musical Instrument Sounds](https://mlsanity.com/paper/2208.06169/) (2022)
- [Neural Waveshaping Synthesis](https://mlsanity.com/paper/2107.05050/) (2021)
- [A Review of Intelligent Music Generation Systems](https://mlsanity.com/paper/2211.09124/) (2022)
- [Music Style Transfer with Time-Varying Inversion of Diffusion Models](https://mlsanity.com/paper/2402.13763/) (2024)
- [Audio representations for deep learning in sound synthesis: A review](https://mlsanity.com/paper/2201.02490/) (2022)
- [Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfer](https://mlsanity.com/paper/2409.06096/) (2024)
- [Timbre Transfer with Variational Auto Encoding and Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/2109.02096/) (2021)
- [Self-Supervised Audio-Visual Soundscape Stylization](https://mlsanity.com/paper/2409.14340/) (2024)
- [Streamable Neural Audio Synthesis With Non-Causal Convolutions](https://mlsanity.com/paper/2204.07064/) (2022)
- [Timbre transfer using image-to-image denoising diffusion implicit models](https://mlsanity.com/paper/2307.04586/) (2023)
- [DarkGAN: Exploiting Knowledge Distillation for Comprehensible Audio Synthesis with GANs](https://mlsanity.com/paper/2108.01216/) (2021)
- [AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfer](https://mlsanity.com/paper/2603.15597/) (2026)
- [SynthScribe: Deep Multimodal Tools for Synthesizer Sound Retrieval and Exploration](https://mlsanity.com/paper/2312.04690/) (2023)
- [Real-time Timbre Remapping with Differentiable DSP](https://mlsanity.com/paper/2407.04547/) (2024)
- [PESTO: Real-Time Pitch Estimation with Self-supervised Transposition-equivariant Objective](https://mlsanity.com/paper/2508.01488/) (2025)
- [Diffusion Timbre Transfer Via Mutual Information Guided Inpainting](https://mlsanity.com/paper/2601.01294/) (2026)
- [WaveTransfer: A Flexible End-to-end Multi-instrument Timbre Transfer with Diffusion](https://mlsanity.com/paper/2409.15321/) (2024)
- [Musical Speech: A Transformer-based Composition Tool](https://mlsanity.com/paper/2108.01043/) (2021)
- [CAESynth: Real-Time Timbre Interpolation and Pitch Control with Conditional Autoencoders](https://mlsanity.com/paper/2111.05174/) (2021)
- [Sound Design Strategies for Latent Audio Space Explorations Using Deep Learning Architectures](https://mlsanity.com/paper/2305.15571/) (2023)
- [Repurposing Image Diffusion Models for Training-Free Music Style Transfer on Mel-spectrograms](https://mlsanity.com/paper/2411.15913/) (2024)
- [Revisiting Your Memory: Reconstruction of Affect-Contextualized Memory via EEG-guided Audiovisual Generation](https://mlsanity.com/paper/2412.05296/) (2024)
- [Designing Neural Synthesizers for Low-Latency Interaction](https://mlsanity.com/paper/2503.11562/) (2025)
- [GuitarFlow: Realistic Electric Guitar Synthesis From Tablatures via Flow Matching and Style Transfer](https://mlsanity.com/paper/2510.21872/) (2025)
- [Music Transcription with (Almost) No Supervision](https://mlsanity.com/paper/2605.24193/) (2026)
- [Can Machines Generate Personalized Music? A Hybrid Favorite-aware Method for User Preference Music Transfer](https://mlsanity.com/paper/2201.08526/) (2022)
- [Music Style Transfer With Diffusion Model](https://mlsanity.com/paper/2404.14771/) (2024)

## Abstract (arXiv)

In this work, we address the problem of musical timbre transfer, where the goal is to manipulate the timbre of a sound sample from one instrument to match another instrument while preserving other musical content, such as pitch, rhythm, and loudness. In principle, one could apply image-based style transfer techniques to a time-frequency representation of an audio signal, but this depends on having a representation that allows independent manipulation of timbre as well as high-quality waveform generation. We introduce TimbreTron, a method for musical timbre transfer which applies "image" domain style transfer to a time-frequency representation of the audio signal, and then produces a high-quality waveform using a conditional WaveNet synthesizer. We show that the Constant Q Transform (CQT) representation is particularly well-suited to convolutional architectures due to its approximate pitch equivariance. Based on human perceptual evaluations, we confirmed that TimbreTron recognizably transferred the timbre while otherwise preserving the musical content, for both monophonic and polyphonic samples.