# MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models

arXiv: https://arxiv.org/abs/2402.06178 · HTML: https://mlsanity.com/paper/2402.06178/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 40 of its 50 references that are in the map.
- First posted on arXiv 9 Feb 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 40)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model](https://mlsanity.com/paper/2304.13731/) (2023)
- [Mustango: Toward Controllable Text-to-Music Generation](https://mlsanity.com/paper/2311.08355/) (2023)
- [MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies](https://mlsanity.com/paper/2308.01546/) (2023)
- [POP909: A Pop-song Dataset for Music Arrangement Generation](https://mlsanity.com/paper/2008.07142/) (2020)
- [AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models](https://mlsanity.com/paper/2304.00830/) (2023)
- [Music ControlNet: Multiple Time-varying Controls for Music Generation](https://mlsanity.com/paper/2311.07069/) (2023)
- [Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning](https://mlsanity.com/paper/2308.11276/) (2023)
- [DITTO: Diffusion Inference-Time T-Optimization for Music Generation](https://mlsanity.com/paper/2401.12179/) (2024)
- [The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation](https://mlsanity.com/paper/2311.10057/) (2023)
- [Instruct-Imagen: Image Generation with Multi-modal Instruction](https://mlsanity.com/paper/2401.01952/) (2024)
- [MuseCoco: Generating Symbolic Music from Text](https://mlsanity.com/paper/2306.00110/) (2023)
- [Efficient Neural Music Generation](https://mlsanity.com/paper/2305.15719/) (2023)
- [JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models](https://mlsanity.com/paper/2308.04729/) (2023)
- [HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks](https://mlsanity.com/paper/2006.05694/) (2020)

## Built on by (most cited first, 30 of 32)

- [Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversion](https://mlsanity.com/paper/2402.10009/) (2024)
- [Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models](https://mlsanity.com/paper/2406.08384/) (2024)
- [Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning](https://mlsanity.com/paper/2405.18386/) (2024)
- [High Fidelity Text-Guided Music Editing via Single-Stage Flow Matching](https://mlsanity.com/paper/2407.03648/) (2024)
- [Editing Music with Melody and Text: Using ControlNet for Diffusion Transformer](https://mlsanity.com/paper/2410.05151/) (2024)
- [Audio Prompt Adapter: Unleashing Music Editing Abilities for Text-to-Music with Lightweight Finetuning](https://mlsanity.com/paper/2407.16564/) (2024)
- [MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music Generation](https://mlsanity.com/paper/2407.15060/) (2024)
- [Improving Musical Accompaniment Co-creation via Diffusion Transformers](https://mlsanity.com/paper/2410.23005/) (2024)
- [Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfer](https://mlsanity.com/paper/2409.06096/) (2024)
- [AudioMorphix: Training-free audio editing with diffusion probabilistic models](https://mlsanity.com/paper/2505.16076/) (2025)
- [SonicMaster: Towards Controllable All-in-One Music Restoration and Mastering](https://mlsanity.com/paper/2508.03448/) (2025)
- [Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation](https://mlsanity.com/paper/2407.20445/) (2024)
- [TADA! Tuning Audio Diffusion Models through Activation Steering](https://mlsanity.com/paper/2602.11910/) (2026)
- [SoundMorpher: Perceptually-Uniform Sound Morphing with Diffusion Model](https://mlsanity.com/paper/2410.02144/) (2024)
- [LiLAC: A Lightweight Latent ControlNet for Musical Audio Generation](https://mlsanity.com/paper/2506.11476/) (2025)
- [GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Models](https://mlsanity.com/paper/2506.17886/) (2025)
- [Via Score to Performance: Efficient Human-Controllable Long Song Generation with Bar-Level Symbolic Notation](https://mlsanity.com/paper/2508.01394/) (2025)
- [Unsupervised Disentanglement of Content and Style via Variance-Invariance Constraints](https://mlsanity.com/paper/2407.03824/) (2024)
- [WhisQ: Cross-Modal Representation Learning for Text-to-Music MOS Prediction](https://mlsanity.com/paper/2506.05899/) (2025)
- [Neural Speech Extraction with Human Feedback](https://mlsanity.com/paper/2508.03041/) (2025)
- [The Interpretation Gap in Text-to-Music Generation Models](https://mlsanity.com/paper/2407.10328/) (2024)
- [Prevailing Research Areas for Music AI in the Era of Foundation Models](https://mlsanity.com/paper/2409.09378/) (2024)
- [RASA: Replace Anyone, Say Anything -- A Training-Free Framework for Audio-Driven and Universal Portrait Video Editing](https://mlsanity.com/paper/2503.11571/) (2025)
- [Not that Groove: Zero-Shot Symbolic Music Editing](https://mlsanity.com/paper/2505.08203/) (2025)
- [Noise-to-Notes: Diffusion-based Generation and Refinement for Automatic Drum Transcription](https://mlsanity.com/paper/2509.21739/) (2025)
- [AI-Assisted Music Production: A User Study on Text-to-Music Models](https://mlsanity.com/paper/2509.23364/) (2025)
- [MIDI-LLM: Improving Text-to-MIDI Music Generation via Adapting Large Language Models](https://mlsanity.com/paper/2511.03942/) (2025)
- [Annotation-Free MIDI-to-Audio Synthesis via Concatenative Synthesis and Generative Refinement](https://mlsanity.com/paper/2410.16785/) (2024)
- [Mozualization: Crafting Music and Visual Representation with Multimodal AI](https://mlsanity.com/paper/2504.13891/) (2025)
- [BNMusic: Blending Environmental Noises into Personalized Music](https://mlsanity.com/paper/2506.10754/) (2025)

## Abstract (arXiv)

Recent advances in text-to-music generation models have opened new avenues in musical creativity. However, music generation usually involves iterative refinements, and how to edit the generated music remains a significant challenge. This paper introduces a novel approach to the editing of music generated by such models, enabling the modification of specific attributes, such as genre, mood and instrument, while maintaining other aspects unchanged. Our method transforms text editing to textitlatent space manipulation while adding an extra constraint to enforce consistency. It seamlessly integrates with existing pretrained text-to-music diffusion models without requiring additional training. Experimental results demonstrate superior performance over both zero-shot and certain supervised baselines in style and timbre transfer evaluations. Additionally, we showcase the practical applicability of our approach in real-world music editing scenarios.