mlsanityData to 14 Oct 2026

WaveTransfer: A Flexible End-to-end Multi-instrument Timbre Transfer with Diffusion

arXiv 2409.15321 ↗First posted 6 Sep 2024eess.AS cs.SDcited by 2

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

As diffusion-based deep generative models gain prevalence, researchers are actively investigating their potential applications across various domains, including music synthesis and style alteration. Within this work, we are interested in timbre transfer, a process that involves seamlessly altering the instrumental characteristics of musical pieces while preserving essential musical elements. This paper introduces WaveTransfer, an end-to-end diffusion model designed for timbre transfer. We specifically employ the bilateral denoising diffusion model (BDDM) for noise scheduling search. Our model is capable of conducting timbre transfer between audio mixtures as well as individual instruments. Notably, it exhibits versatility in that it accommodates multiple types of timbre transfer between unique instrument pairs in a single model, eliminating the need for separate model training for each pairing. Furthermore, unlike recent works limited to 16 kHz, WaveTransfer can be trained at various sampling rates, including the industry-standard 44.1 kHz, a feature of particular interest to the music community.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 22 of its 35 references that are in the map. First posted on arXiv 6 Sep 2024.

Builds on

22 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Timbre transfer using image-to-image denoising diffusion implicit models10 Jul
  2. 2022BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis25 Mar

Other references · 20

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Denoising Diffusion Implicit Modelscited by 9,781
  3. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  4. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  5. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  6. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  7. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  8. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 225
  9. 2023Adapting Frechet Audio Distance for Generative Music Evaluationcited by 92
  10. 2020ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metriccited by 79
  11. 2019Encoding Musical Style with Transformer Autoencoderscited by 35
  12. 2018TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfercited by 33
8 more
  1. 2018Symbolic Music Genre Transfer with CycleGANcited by 28
  2. 2019Learning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoderscited by 21
  3. 2022MusIAC: An extensible generative framework for Music Infilling Applications with multi-level Controlcited by 17
  4. 2021Self-Supervised VQ-VAE for One-Shot Music Style Transfercited by 14
  5. 2020Hierarchical Timbre-Painting and Articulation Generationcited by 9
  6. 2018Modulated Variational auto-Encoders for many-to-many musical timbre transfercited by 8
  7. 2021Effective Low-Cost Time-Domain Audio Separation Using Globally Attentive Locally Recurrent Networkscited by 8
  8. 2023AQUATK: An Audio Quality Assessment Toolkitcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Diffusion Timbre Transfer Via Mutual Information Guided Inpaintingcited by 3
  2. 2026Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Combining audio control and style transfer using latent diffusion
  2. 2024Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfer
  3. 2025Diff-TONE: Timestep Optimization for iNstrument Editing in Text-to-Music Diffusion Models
  4. 2024Subtractive Training for Music Stem Insertion using Latent Diffusion Models
  5. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models
  6. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformers
  7. 2026Research on Piano Timbre Transformation System Based on Diffusion Model
  8. 2025Generation of Musical Timbres using a Text-Guided Diffusion Model