mlsanityData to 7 Oct 2026

Synthesis and editing of multi-instrument audio mixtures using scalar-quantised latents with MIDI Span conditioning

arXiv 2609.25546 ↗First posted 22 Sep 2026cs.SD cs.LG eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

Music creation often involves iterative refinement, changing selected musical details while retaining the rest. To support such refinement, we introduce SpanSynth-Edit, a flow-matching model for MIDI-guided synthesis and editing of multi-instrument audio mixtures using low-frame-rate scalar-quantised latents. MIDI Span encodes instrument-labelled note lifecycles as unordered event sets with continuous-valued attributes and pools each set into one conditioning vector per audio-latent frame. The model uses contextual audio for instrument-specific timbre guidance and supports editing by resynthesising the target region from revised MIDI. Experiments on single- and multi-instrument benchmarks show competitive performance and demonstrate within-frame onset control. We also discuss limitations of transcription-based note-adherence evaluation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 51 references that are in the map. First posted on arXiv 22 Sep 2026.

Builds on

33 of 51 references.

References · 33

  1. 2022Scalable Diffusion Models with Transformerscited by 5,931
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2017Deep Setscited by 1,394
  4. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  5. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  6. 2018Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Datasetcited by 215
  7. 2024FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Modelscited by 214
  8. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  9. 2019Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantitycited by 86
  10. 2020POP909: A Pop-song Dataset for Music Arrangement Generationcited by 83
  11. 2021MT3: Multi-Task Multitrack Music Transcriptioncited by 70
  12. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 66
21 more
  1. 2016Learning Features of Music from Scratchcited by 57
  2. 2022Multi-instrument Music Synthesis with Spectrogram Diffusioncited by 42
  3. 2022Unaligned Supervision For Automatic Music Transcription in The Wildcited by 25
  4. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  5. 2024Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuningcited by 15
  6. 2024High Fidelity Text-Guided Music Editing via Single-Stage Flow Matchingcited by 13
  7. 2024YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentationcited by 13
  8. 2026HeartMuLa: A Family of Open Sourced Music Foundation Modelscited by 13
  9. 2024Combining audio control and style transfer using latent diffusioncited by 10
  10. 2025MIDI-VALLE: Improving Expressive Piano Performance Synthesis Through Neural Codec Language Modellingcited by 8
  11. 2025Unified Cross-modal Translation of Score Images, Symbolic Music, and Performance Audiocited by 6
  12. 2025GOAT: A Large Dataset of Paired Guitar Audio Recordings and Tablaturescited by 6
  13. 2026MuQ-Eval: An Open-Source Per-Sample Quality Metric for AI Music Generation Evaluationcited by 6
  14. 2025TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrumentcited by 5
  15. 2025PianoVAM: A Multimodal Piano Performance Datasetcited by 5
  16. 2023FiloBass: A Dataset and Corpus Based Study of Jazz Basslinescited by 3
  17. 2024Leveraging Real Electric Guitar Tones and Effects to Improve Robustness in Guitar Tablature Transcription Modelingcited by 3
  18. 2025ImprovNet -- Generating Controllable Musical Improvisations with Iterative Corruption Refinementcited by 3
  19. 2025SyMuPe: Affective and Controllable Symbolic Music Performancecited by 3
  20. 2026Diffusion Timbre Transfer Via Mutual Information Guided Inpaintingcited by 3
  21. 2026P-MUSE: Prompt-MIDI-Optional Model for Unified Instrumental Music Synthesis and Editingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance
  2. 2025FlowSynth: Instrument Generation Through Distributional Flow Matching and Test-Time Search
  3. 2023InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Models
  4. 2026FlowSonic: Stable Zero-Shot Music Editing via High-Order Trajectory Integration
  5. 2026DiffSynth-Music: Audio-Conditioned KV-Cache Adapters for Controllable Music Generation
  6. 2025Diff-TONE: Timestep Optimization for iNstrument Editing in Text-to-Music Diffusion Models
  7. 2025MusicWeaver: Programmable Long-Form Music Generation with Provably Local Editing ↗
  8. 2026MuScriptor: An Open Model for Multi-Instrument Music Transcription