mlsanityData to 7 Oct 2026

Subtractive Training for Music Stem Insertion using Latent Diffusion Models

arXiv 2406.19328 ↗First posted 27 Jun 2024cs.SD cs.LG eess.AScited by 2

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We present Subtractive Training, a simple and novel method for synthesizing individual musical instrument stems given other instruments as context. This method pairs a dataset of complete music mixes with 1) a variant of the dataset lacking a specific stem, and 2) LLM-generated instructions describing how the missing stem should be reintroduced. We then fine-tune a pretrained text-to-audio diffusion model to generate the missing instrument stem, guided by both the existing stems and the text instruction. Our results demonstrate Subtractive Training's efficacy in creating authentic drum stems that seamlessly blend with the existing tracks. We also show that we can use the text instruction to control the generation of the inserted stem in terms of rhythm, dynamics, and genre, allowing us to modify the style of a single instrument in a full song while keeping the remaining instruments the same. Lastly, we extend this technique to MIDI formats, successfully generating compatible bass, drum, and guitar parts for incomplete arrangements.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 34 of its 41 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

34 of 41 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models12 Jun
  2. 2023InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Models28 Aug
  3. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies3 Aug
  4. 2023Generating symbolic music using diffusion models15 Mar
  5. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models8 Feb
  6. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan

Other references · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  9. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2019An Introduction to Variational Autoencoderscited by 997
  12. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
16 more
  1. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  2. 2020Jukebox: A Generative Model for Musiccited by 535
  3. 2023Simple and Controllable Music Generationcited by 480
  4. 2023MusicLM: Generating Music From Textcited by 442
  5. 2019Image inpainting: A reviewcited by 89
  6. 2023LP-MusicCaps: LLM-Based Pseudo Music Captioningcited by 83
  7. 2023Music ControlNet: Multiple Time-varying Controls for Music Generationcited by 82
  8. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  9. 2019Demucs: Deep Extractor for Music Sources with extra unlabeled data remixedcited by 53
  10. 2023A Comprehensive Survey on Knowledge Distillation of Diffusion Modelscited by 45
  11. 2024Joint Audio and Symbolic Conditioning for Temporally Controlled Text-to-Music Generationcited by 27
  12. 2021DadaGP: A Dataset of Tokenized GuitarPro Songs for Sequence Modelscited by 20
  13. 2023Content-based Controls For Music Large Language Modelingcited by 19
  14. 2024Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation and Editing via Content-based Controlscited by 17
  15. 2022Evaluating generative audio systems and their metricscited by 16
  16. 2024Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026DARC: Drum accompaniment generation with fine-grained rhythm controlcited by 2
  2. 2026Latent Fourier Transformcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024WaveTransfer: A Flexible End-to-end Multi-instrument Timbre Transfer with Diffusion
  2. 2025Diff-TONE: Timestep Optimization for iNstrument Editing in Text-to-Music Diffusion Models
  3. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformers
  4. 2024Bass Accompaniment Generation via Latent Diffusion
  5. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Model
  6. 2025Generation of Musical Timbres using a Text-Guided Diffusion Model
  7. 2026Diffusion Timbre Transfer Via Mutual Information Guided Inpainting
  8. 2026Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems