mlsanityData to 14 Oct 2026

Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversion

arXiv 2402.10009 ↗First posted 15 Feb 2024cs.SD cs.LG eess.AScited by 60

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Editing signals using large pre-trained models, in a zero-shot manner, has recently seen rapid advancements in the image domain. However, this wave has yet to reach the audio domain. In this paper, we explore two zero-shot editing techniques for audio signals, which use DDPM inversion with pre-trained diffusion models. The first, which we coin ZEro-shot Text-based Audio (ZETA) editing, is adopted from the image domain. The second, named ZEro-shot UnSupervized (ZEUS) editing, is a novel approach for discovering semantically meaningful editing directions without supervision. When applied to music signals, this method exposes a range of musically interesting modifications, from controlling the participation of specific instruments to improvisations on the melody. Samples and code can be found in https://hilamanor.github.io/AudioEditing/ .

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 47 of its 59 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

47 of 59 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models9 Feb
  2. 2023Investigating Personalization Methods in Text to Music Generation20 Sep
  3. 2023InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Models28 Aug
  4. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  5. 2023AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models3 Apr
  6. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan

Other references · 41

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
29 more
  1. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  2. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  3. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  5. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  6. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
  7. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  8. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  9. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 695
  10. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  11. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 642
  12. 2023MusicLM: Generating Music From Textcited by 448
  13. 2020Closed-Form Factorization of Latent Semantics in GANscited by 408
  14. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 332
  15. 2022Diffusion Models already have a Semantic Latent Spacecited by 328
  16. 2023An Edit Friendly DDPM Noise Space: Inversion and Manipulationscited by 244
  17. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 225
  18. 2016FMA: A Dataset For Music Analysiscited by 207
  19. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 176
  20. 2022SINE: SINgle Image Editing with Text-to-Image Diffusion Modelscited by 159
  21. 2021Taming Visually Guided Sound Generationcited by 136
  22. 2023Adapting Frechet Audio Distance for Generative Music Evaluationcited by 92
  23. 2023Discovering Interpretable Directions in the Semantic Latent Space of Diffusion Modelscited by 60
  24. 2023Training-free Content Injection using h-space in Diffusion Modelscited by 36
  25. 2023On the Posterior Distribution in Denoising: Application to Uncertainty Quantificationcited by 33
  26. 2023Unsupervised Discovery of Semantic Latent Directions in Diffusion Modelscited by 28
  27. 2023Uncertainty Quantification via Neural Posterior Principal Componentscited by 16
  28. 2022SDMuse: Stochastic Differential Music Editing and Generation via Hybrid Representationcited by 11
  29. 2023Audio Editing with Non-Rigid Text Promptscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 26; 2026: 17.

60 papers.

Sort:

By area:

  1. 2024FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Modelscited by 217
  2. 2024Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributionscited by 53
  3. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Modelscited by 33
  4. 2025Compressed Image Generation with Denoising Diffusion Codebook Modelscited by 28
  5. 2024Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuningcited by 15
  6. 2024Exploring Low-Dimensional Subspaces in Diffusion Models for Controllable Image Editingcited by 15
  7. 2024Free Hunch: Denoiser Covariance Estimation for Diffusion Models Without Extra Costscited by 14
  8. 2025Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoisingcited by 14
  9. 2025Object-AVEdit: An Object-level Audio-Visual Editing Modelcited by 14
  10. 2024High Fidelity Text-Guided Music Editing via Single-Stage Flow Matchingcited by 13
  11. 2024Adaptive Compressed Sensing with Diffusion-Based Posterior Samplingcited by 13
  12. 2024AudioEditor: A Training-Free Diffusion-Based Audio Editing Frameworkcited by 13
38 more
  1. 2025Guiding Audio Editing with Audio Language Modelcited by 12
  2. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editingcited by 12
  3. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformerscited by 11
  4. 2024SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generationcited by 9
  5. 2024Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfercited by 9
  6. 2025AVI-Edit: Audio-sync Video Instance Editing with Granularity-Aware Mask Refinercited by 9
  7. 2024MEDIC: Zero-shot Music Editing with Disentangled Inversion Controlcited by 8
  8. 2025Turbo-DDCM: Fast and Flexible Zero-Shot Diffusion-Based Image Compressioncited by 8
  9. 2025AudioMorphix: Training-free audio editing with diffusion probabilistic modelscited by 7
  10. 2025Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Editscited by 7
  11. 2025SonicMaster: Towards Controllable All-in-One Music Restoration and Masteringcited by 6
  12. 2025XAttnMark: Learning Robust Audio Watermarking with Cross-Attentioncited by 5
  13. 2026Spanning the Visual Analogy Space with a Weight Basis of LoRAscited by 5
  14. 2026MMAE: A Massive Multitask Audio Editing Benchmarkcited by 5
  15. 2026Improving Robotic Generalist Policies via Flow Reversal Steeringcited by 5
  16. 2025ZeroSep: Separate Anything in Audio with Zero Trainingcited by 4
  17. 2025SAO-Instruct: Free-form Audio Editing using Natural Language Instructionscited by 4
  18. 2025FreeSliders: Training-Free, Modality-Agnostic Concept Sliders for Fine-Grained Diffusion Control in Images, Audio, and Videocited by 4
  19. 2025Schrodinger Audio-Visual Editor: Object-Level Audiovisual Removalcited by 4
  20. 2026TADA! Tuning Audio Diffusion Models through Activation Steeringcited by 4
  21. 2024Gaussian Flow Bridges for Audio Domain Transfer with Unpaired Datacited by 3
  22. 2024SoundMorpher: Perceptually-Uniform Sound Morphing with Diffusion Modelcited by 3
  23. 2025LiLAC: A Lightweight Latent ControlNet for Musical Audio Generationcited by 3
  24. 2025RFM-Editing: Rectified Flow Matching for Text-guided Audio Editingcited by 3
  25. 2025FlowOpt: Fast Optimization Through Whole Flow Processes for Training-Free Editingcited by 3
  26. 2025Melodia: Training-Free Music Editing Guided by Attention Probing in Diffusion Modelscited by 3
  27. 2026UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusioncited by 2
  28. 2024Shallow Diffuse: Robust and Invisible Watermarking through Low-Dimensional Subspaces in Diffusion Modelscited by 1
  29. 2025A Simple Combination of Diffusion Models for Better Quality Trade-Offs in Image Denoisingcited by 1
  30. 2025Evaluating Music Context Preservation: A Multi-facet Framework for Music Editing Systemscited by 1
  31. 2026DirectAudioEdit: Inversion-Free Text-Guided Audio Editing via Diffusion Prediction Contrastcited by 1
  32. 2026FreeSonic: Training-Free Temporal-Aware Decoupled Attention for Precise Audio Editingcited by 1
  33. 2024Annotation-Free MIDI-to-Audio Synthesis via Concatenative Synthesis and Generative Refinementcited by 0
  34. 2025BNMusic: Blending Environmental Noises into Personalized Musiccited by 0
  35. 2025Abstract Sound Fusion with Unconditional Inversion Modelscited by 0
  36. 2025EditGen: Harnessing Cross-Attention Control for Instruction-Based Auto-Regressive Audio Editingcited by 0
  37. 2025Audio-Guided Visual Editing with Complex Multi-Modal Promptscited by 0
  38. 2025Virtual Consistency for Audio Editingcited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SteerMusic: Enhanced Musical Consistency for Zero-shot Text-guided and Personalized Music Editing
  2. 2024Editing Music with Melody and Text: Using ControlNet for Diffusion Transformer
  3. 2026Polyphonia: Zero-Shot Timbre Transfer in Polyphonic Music with Acoustic-Informed Attention Calibration ↗
  4. 2024Prompt-guided Precise Audio Editing with Diffusion Models
  5. 2025Diff-TONE: Timestep Optimization for iNstrument Editing in Text-to-Music Diffusion Models
  6. 2021A Unified Model for Zero-shot Music Source Separation, Transcription and Synthesis
  7. 2026Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance
  8. 2026Diffusion Timbre Transfer Via Mutual Information Guided Inpainting