mlsanityData to 7 Oct 2026

Audio Prompt Adapter: Unleashing Music Editing Abilities for Text-to-Music with Lightweight Finetuning

arXiv 2407.16564 ↗First posted 23 Jul 2024cs.SD cs.AI eess.AScited by 12

Not on the map. Why?

Abstract (arXiv)

Text-to-music models allow users to generate nearly realistic musical audio with textual commands. However, editing music audios remains challenging due to the conflicting desiderata of performing fine-grained alterations on the audio while maintaining a simple user interface. To address this challenge, we propose Audio Prompt Adapter (or AP-Adapter), a lightweight addition to pretrained text-to-music models. We utilize AudioMAE to extract features from the input audio, and construct attention-based adapters to feedthese features into the internal layers of AudioLDM2, a diffusion-based text-to-music model. With 22M trainable parameters, AP-Adapter empowers users to harness both global (e.g., genre and timbre) and local (e.g., melody) aspects of music, using the original audio and a short text as inputs. Through objective and subjective studies, we evaluate AP-Adapter on three tasks: timbre transfer, genre transfer, and accompaniment generation. Additionally, we demonstrate its effectiveness on out-of-domain audios containing unseen instruments during training.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 30 of its 39 references that are in the map. First posted on arXiv 23 Jul 2024.

Builds on

30 of 39 references.

References · 30

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  9. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  10. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  11. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  12. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
18 more
  1. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  2. 2023Simple and Controllable Music Generationcited by 480
  3. 2023MusicLM: Generating Music From Textcited by 442
  4. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  5. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 221
  6. 2023Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Modelscited by 203
  7. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  8. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
  9. 2023Stay on topic with Classifier-Free Guidancecited by 90
  10. 2023Music ControlNet: Multiple Time-varying Controls for Music Generationcited by 82
  11. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Modelscited by 32
  12. 2020AI Song Contest: Human-AI Co-Creation in Songwritingcited by 25
  13. 2023M2UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Modelscited by 24
  14. 2023InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Modelscited by 23
  15. 2023Content-based Controls For Music Large Language Modelingcited by 19
  16. 2023Investigating Personalization Methods in Text to Music Generationcited by 11
  17. 2024MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music Generationcited by 11
  18. 2021Expressive Communication: A Common Framework for Evaluating Developments in Generative Models and Steering Interfacescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 5.

12 papers.

  1. 2025MuseControlLite: Multifunctional Music Generation with Lightweight Conditionerscited by 9
  2. 2025A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectivescited by 4
  3. 2025Not that Groove: Zero-Shot Symbolic Music Editingcited by 1
  4. 2025MIDI-LLM: Improving Text-to-MIDI Music Generation via Adapting Large Language Modelscited by 1
  5. 2025Evaluating Music Context Preservation: A Multi-facet Framework for Music Editing Systemscited by 1
  6. 2026Stemphonic: All-at-once Flexible Multi-stem Music Generationcited by 1
  7. 2026Low-Resource Guidance for Controllable Latent Audio Diffusioncited by 1
  8. 2026UAT: Unified Audio-Text Diffusion for Audio Generation, Editing, and Captioningcited by 1
  9. 2025Virtual Consistency for Audio Editingcited by 0
  10. 2025MusRec: Zero-Shot Text-to-Music Editing via Rectified Flow and Diffusion Transformerscited by 0
  11. 2026AnchorSteer: Self-Discovered Concept Injection for Structure-Preserving Music Editingcited by 0
  12. 2026FlowSonic: Stable Zero-Shot Music Editing via High-Order Trajectory Integrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Melodia: Training-Free Music Editing Guided by Attention Probing in Diffusion Models
  2. 2024Editing Music with Melody and Text: Using ControlNet for Diffusion Transformer
  3. 2025Diff-TONE: Timestep Optimization for iNstrument Editing in Text-to-Music Diffusion Models
  4. 2026S2Accompanist: A Semantic-Aware and Structure-Guided Diffusion Model for Music Accompaniment Generation
  5. 2024Prompt-guided Precise Audio Editing with Diffusion Models
  6. 2026Instrumental Text-to-Music Generation with Auxiliary Conditioning Branches ↗
  7. 2024Combining audio control and style transfer using latent diffusion
  8. 2026TADA! Tuning Audio Diffusion Models through Activation Steering