mlsanityData to 7 Oct 2026

Diffusion Timbre Transfer Via Mutual Information Guided Inpainting

arXiv 2601.01294 ↗First posted 3 Jan 2026cs.SD cs.AI eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

We study timbre transfer as an inference-time editing problem for music audio. Starting from a strong pre-trained latent diffusion model, we introduce a lightweight procedure that requires no additional training: (i) a dimension-wise noise injection that targets latent channels most informative of instrument identity, and (ii) an early-step clamping mechanism that re-imposes the input's melodic and rhythmic structure during reverse diffusion. The method operates directly on audio latents and is compatible with text/audio conditioning (e.g., CLAP). We discuss design choices,analyze trade-offs between timbral change and structural preservation, and show that simple inference-time controls can meaningfully steer pre-trained models for style-transfer use cases.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 32 of its 34 references that are in the map. First posted on arXiv 3 Jan 2026.

Builds on

32 of 34 references.

References · 32

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Denoising Diffusion Implicit Modelscited by 9,655
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  5. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  6. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  7. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  8. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
  9. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
  10. 2020DDSP: Differentiable Digital Signal Processingcited by 177
  11. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  12. 2021Symbolic Music Generation with Diffusion Modelscited by 152
20 more
  1. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speechcited by 147
  2. 2021Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Schemecited by 99
  3. 2023Music ControlNet: Multiple Time-varying Controls for Music Generationcited by 82
  4. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  5. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Modelscited by 33
  6. 2024Music2Latent: Consistency Autoencoders for Latent Audio Compressioncited by 33
  7. 2018TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfercited by 32
  8. 2023Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptationcited by 30
  9. 2024DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generationcited by 26
  10. 2025Ambient Diffusion Omni: Training Good Models with Bad Datacited by 18
  11. 2024Editing Music with Melody and Text: Using ControlNet for Diffusion Transformercited by 13
  12. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformerscited by 11
  13. 2024Music Style Transfer with Time-Varying Inversion of Diffusion Modelscited by 9
  14. 2024Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfercited by 9
  15. 2018Modulated Variational auto-Encoders for many-to-many musical timbre transfercited by 8
  16. 2021Timbre Transfer with Variational Auto Encoding and Cycle-Consistent Adversarial Networkscited by 8
  17. 2023Timbre transfer using image-to-image denoising diffusion implicit modelscited by 7
  18. 2025LiLAC: A Lightweight Latent ControlNet for Musical Audio Generationcited by 3
  19. 2025PESTO: Real-Time Pitch Estimation with Self-supervised Transposition-equivariant Objectivecited by 3
  20. 2024WaveTransfer: A Flexible End-to-end Multi-instrument Timbre Transfer with Diffusioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026TADA! Tuning Audio Diffusion Models through Activation Steeringcited by 2
  2. 2026PianoKontext: Expressive Performance Rendering from Deadpan Contextcited by 0
  3. 2026Synthesis and editing of multi-instrument audio mixtures using scalar-quantised latents with MIDI Span conditioningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Combining audio control and style transfer using latent diffusion
  2. 2025Diff-TONE: Timestep Optimization for iNstrument Editing in Text-to-Music Diffusion Models
  3. 2026Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems
  4. 2023InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Models
  5. 2026Audio-to-Audio via Diffusion Warm Initialization ↗
  6. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models
  7. 2026Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probes
  8. 2025Generation of Musical Timbres using a Text-Guided Diffusion Model