# Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversion

arXiv: https://arxiv.org/abs/2402.10009 · HTML: https://mlsanity.com/paper/2402.10009/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 47 of its 59 references that are in the map.
- First posted on arXiv 15 Feb 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 47)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [Closed-Form Factorization of Latent Semantics in GANs](https://mlsanity.com/paper/2007.06600/) (2020)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation](https://mlsanity.com/paper/2011.12799/) (2020)
- [Diffusion Models already have a Semantic Latent Space](https://mlsanity.com/paper/2210.10960/) (2022)
- [An Edit Friendly DDPM Noise Space: Inversion and Manipulations](https://mlsanity.com/paper/2304.06140/) (2023)

## Built on by (most cited first, 30 of 60)

- [FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Models](https://mlsanity.com/paper/2412.08629/) (2024)
- [Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributions](https://mlsanity.com/paper/2409.02426/) (2024)
- [Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models](https://mlsanity.com/paper/2406.08384/) (2024)
- [Compressed Image Generation with Denoising Diffusion Codebook Models](https://mlsanity.com/paper/2502.01189/) (2025)
- [Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning](https://mlsanity.com/paper/2405.18386/) (2024)
- [Exploring Low-Dimensional Subspaces in Diffusion Models for Controllable Image Editing](https://mlsanity.com/paper/2409.02374/) (2024)
- [Free Hunch: Denoiser Covariance Estimation for Diffusion Models Without Extra Costs](https://mlsanity.com/paper/2410.11149/) (2024)
- [Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoising](https://mlsanity.com/paper/2503.20782/) (2025)
- [Object-AVEdit: An Object-level Audio-Visual Editing Model](https://mlsanity.com/paper/2510.00050/) (2025)
- [High Fidelity Text-Guided Music Editing via Single-Stage Flow Matching](https://mlsanity.com/paper/2407.03648/) (2024)
- [Adaptive Compressed Sensing with Diffusion-Based Posterior Sampling](https://mlsanity.com/paper/2407.08256/) (2024)
- [AudioEditor: A Training-Free Diffusion-Based Audio Editing Framework](https://mlsanity.com/paper/2409.12466/) (2024)
- [Guiding Audio Editing with Audio Language Model](https://mlsanity.com/paper/2509.21625/) (2025)
- [Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing](https://mlsanity.com/paper/2604.10708/) (2026)
- [Improving Musical Accompaniment Co-creation via Diffusion Transformers](https://mlsanity.com/paper/2410.23005/) (2024)
- [SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation](https://mlsanity.com/paper/2405.18503/) (2024)
- [Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfer](https://mlsanity.com/paper/2409.06096/) (2024)
- [AVI-Edit: Audio-sync Video Instance Editing with Granularity-Aware Mask Refiner](https://mlsanity.com/paper/2512.10571/) (2025)
- [MEDIC: Zero-shot Music Editing with Disentangled Inversion Control](https://mlsanity.com/paper/2407.13220/) (2024)
- [Turbo-DDCM: Fast and Flexible Zero-Shot Diffusion-Based Image Compression](https://mlsanity.com/paper/2511.06424/) (2025)
- [AudioMorphix: Training-free audio editing with diffusion probabilistic models](https://mlsanity.com/paper/2505.16076/) (2025)
- [Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Edits](https://mlsanity.com/paper/2512.07209/) (2025)
- [SonicMaster: Towards Controllable All-in-One Music Restoration and Mastering](https://mlsanity.com/paper/2508.03448/) (2025)
- [XAttnMark: Learning Robust Audio Watermarking with Cross-Attention](https://mlsanity.com/paper/2502.04230/) (2025)
- [Spanning the Visual Analogy Space with a Weight Basis of LoRAs](https://mlsanity.com/paper/2602.15727/) (2026)
- [MMAE: A Massive Multitask Audio Editing Benchmark](https://mlsanity.com/paper/2606.07229/) (2026)
- [Improving Robotic Generalist Policies via Flow Reversal Steering](https://mlsanity.com/paper/2606.13675/) (2026)
- [ZeroSep: Separate Anything in Audio with Zero Training](https://mlsanity.com/paper/2505.23625/) (2025)
- [SAO-Instruct: Free-form Audio Editing using Natural Language Instructions](https://mlsanity.com/paper/2510.22795/) (2025)
- [FreeSliders: Training-Free, Modality-Agnostic Concept Sliders for Fine-Grained Diffusion Control in Images, Audio, and Video](https://mlsanity.com/paper/2511.00103/) (2025)

## Abstract (arXiv)

Editing signals using large pre-trained models, in a zero-shot manner, has recently seen rapid advancements in the image domain. However, this wave has yet to reach the audio domain. In this paper, we explore two zero-shot editing techniques for audio signals, which use DDPM inversion with pre-trained diffusion models. The first, which we coin ZEro-shot Text-based Audio (ZETA) editing, is adopted from the image domain. The second, named ZEro-shot UnSupervized (ZEUS) editing, is a novel approach for discovering semantically meaningful editing directions without supervision. When applied to music signals, this method exposes a range of musically interesting modifications, from controlling the participation of specific instruments to improvisations on the melody. Samples and code can be found in https://hilamanor.github.io/AudioEditing/ .