# Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems

arXiv: https://arxiv.org/abs/2605.09259 · HTML: https://mlsanity.com/paper/2605.09259/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 10 May 2026.

## Builds on (most cited first, 30 of 31)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms](https://mlsanity.com/paper/1812.08466/) (2018)
- [Stable Audio Open](https://mlsanity.com/paper/2407.14358/) (2024)
- [Hybrid Transformers for Music Source Separation](https://mlsanity.com/paper/2211.08553/) (2022)
- [UniAudio: An Audio Foundation Model Toward Universal Audio Generation](https://mlsanity.com/paper/2310.00704/) (2023)
- [Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme](https://mlsanity.com/paper/2109.13821/) (2021)
- [RAVE: A variational autoencoder for fast and high-quality neural audio synthesis](https://mlsanity.com/paper/2111.05011/) (2021)
- [Multi-Source Diffusion Models for Simultaneous Music Generation and Separation](https://mlsanity.com/paper/2302.02257/) (2023)
- [TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer](https://mlsanity.com/paper/1811.09620/) (2018)
- [Symbolic Music Genre Transfer with CycleGAN](https://mlsanity.com/paper/1809.07575/) (2018)
- [A Universal Music Translation Network](https://mlsanity.com/paper/1805.07848/) (2018)
- [Self-Supervised VQ-VAE for One-Shot Music Style Transfer](https://mlsanity.com/paper/2102.05749/) (2021)
- [The Chamber Ensemble Generator: Limitless High-Quality MIR Data via Generative Modeling](https://mlsanity.com/paper/2209.14458/) (2022)
- [Combining audio control and style transfer using latent diffusion](https://mlsanity.com/paper/2408.00196/) (2024)
- [Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfer](https://mlsanity.com/paper/2409.06096/) (2024)
- [Timbre transfer using image-to-image denoising diffusion implicit models](https://mlsanity.com/paper/2307.04586/) (2023)
- [Prosody-Adaptable Audio Codecs for Zero-Shot Voice Conversion via In-Context Learning](https://mlsanity.com/paper/2505.15402/) (2025)
- [Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversion](https://mlsanity.com/paper/2505.24291/) (2025)
- [Learning Disentangled Representations for Timber and Pitch in Music Audio](https://mlsanity.com/paper/1811.03271/) (2018)
- [DisMix: Disentangling Mixtures of Musical Instruments for Source-level Pitch and Timbre Manipulation](https://mlsanity.com/paper/2408.10807/) (2024)
- [CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance](https://mlsanity.com/paper/2509.19883/) (2025)

## Abstract (arXiv)

Timbre transfer aims to modify the timbral identity of a musical recording while preserving its original melody and rhythm. While single-instrument timbre transfer has made substantial progress, existing approaches to multi-instrument settings commonly rely on separate-then-transfer pipelines that propagate source-separation artifacts and process the resulting stems independently. This paper proposes MixtureTT, to the best of our knowledge the first system for flexible per-stem timbre transfer directly from a polyphonic mixture. Given a mixture and a separate timbre reference for each target voice, MixtureTT jointly transfers all stems to the specified instruments. Its Joint Stem Diffusion Transformer first models cross-stem harmonic context and then refines each output under its own content and timbre conditions. All stems are generated within a single joint diffusion process rather than through separate per-stem diffusion runs. This design avoids propagating waveform-level separation errors through a cascade and produces more coherent multi-stem outputs. Despite operating under a harder input condition, MixtureTT outperforms practical separate-then-transfer pipelines on objective and subjective evaluations, while remaining competitive with single-instrument methods given ground-truth isolated stems. Results on both synthesized and real-audio SATB benchmarks further demonstrate its generalization. Controlled ablations show that jointly modeling cross-stem dependencies improves over an equivalent independent-stem variant, while the structure and timing of this coordination are also critical for effective mixture-level timbre transfer.