# Forging and Removing Latent-Noise Diffusion Watermarks Using a Single Image

arXiv: https://arxiv.org/abs/2504.20111 · HTML: https://mlsanity.com/paper/2504.20111/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 27 Apr 2025.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)
- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 30 of 30)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [The Stable Signature: Rooting Watermarks in Latent Diffusion Models](https://mlsanity.com/paper/2303.15435/) (2023)
- [StegaStamp: Invisible Hyperlinks in Physical Photographs](https://mlsanity.com/paper/1904.05343/) (2019)
- [Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robust](https://mlsanity.com/paper/2305.20030/) (2023)
- [Robust Invisible Video Watermarking with Attention](https://mlsanity.com/paper/1909.01285/) (2019)
- [Gaussian Shading: Provable Performance-Lossless Image Watermarking for Diffusion Models](https://mlsanity.com/paper/2404.04956/) (2024)
- [Invisible Image Watermarks Are Provably Removable Using Generative AI](https://mlsanity.com/paper/2306.01953/) (2023)
- [Watermarking Images in Self-Supervised Latent Spaces](https://mlsanity.com/paper/2112.09581/) (2021)
- [RingID: Rethinking Tree-Ring Watermarking for Enhanced Multi-Key Identification](https://mlsanity.com/paper/2404.14055/) (2024)
- [Robustness of AI-Image Detectors: Fundamental Limits and Practical Attacks](https://mlsanity.com/paper/2310.00076/) (2023)
- [An Undetectable Watermark for Generative Image Models](https://mlsanity.com/paper/2410.07369/) (2024)
- [Frequency-driven Imperceptible Adversarial Attack on Semantic Similarity](https://mlsanity.com/paper/2203.05151/) (2022)
- [Leveraging Optimization for Adaptive Attacks on Image Watermarks](https://mlsanity.com/paper/2309.16952/) (2023)
- [Black-Box Forgery Attacks on Semantic Watermarks for Diffusion Models](https://mlsanity.com/paper/2412.03283/) (2024)
- [Image Watermarks are Removable Using Controllable Regeneration from Clean Noise](https://mlsanity.com/paper/2410.05470/) (2024)
- [Exploring Frequency Adversarial Attacks for Face Forgery Detection](https://mlsanity.com/paper/2203.15674/) (2022)
- [WMAdapter: Adding WaterMark Control to Latent Diffusion Models](https://mlsanity.com/paper/2406.08337/) (2024)
- [A Transfer Attack to Image Watermarks](https://mlsanity.com/paper/2403.15365/) (2024)
- [On the use of automatically generated synthetic image datasets for benchmarking face recognition](https://mlsanity.com/paper/2106.04215/) (2021)
- [Hidden in the Noise: Two-Stage Robust Watermarking for Images](https://mlsanity.com/paper/2412.04653/) (2024)
- [Steganalysis on Digital Watermarking: Is Your Defense Truly Impervious?](https://mlsanity.com/paper/2406.09026/) (2024)
- [Watermark Faker: Towards Forgery of Digital Image Watermarking](https://mlsanity.com/paper/2103.12489/) (2021)

## Built on by (most cited first, 13 of 13)

- [First-Place Solution to NeurIPS 2024 Invisible Watermark Removal Challenge](https://mlsanity.com/paper/2508.21072/) (2025)
- [SLICE: Semantic Latent Injection via Compartmentalized Embedding for Image Watermarking](https://mlsanity.com/paper/2603.12749/) (2026)
- [Secure and Robust Watermarking for AI-generated Images: A Comprehensive Survey](https://mlsanity.com/paper/2510.02384/) (2025)
- [Mitigating Watermark Forgery in Generative Models via Randomized Key Selection](https://mlsanity.com/paper/2507.07871/) (2025)
- [SemBind: Binding Diffusion Watermarks to Semantics Against Black-Box Forgery Attacks](https://mlsanity.com/paper/2601.20310/) (2026)
- [SHIFT: Stochastic Hidden-Trajectory Deflection for Removing Diffusion-based Watermark](https://mlsanity.com/paper/2603.29742/) (2026)
- [ClusterMark: Towards Robust Watermarking for Autoregressive Image Generators with Visual Token Clustering](https://mlsanity.com/paper/2508.06656/) (2025)
- [Towards Robust Content Watermarking Against Removal and Forgery Attacks](https://mlsanity.com/paper/2604.06662/) (2026)
- [MetaSeal: Defending Against Image Attribution Forgery Through Content-Dependent Cryptographic Watermarks](https://mlsanity.com/paper/2509.10766/) (2025)
- [On the Robustness of Watermarking for Autoregressive Image Generation](https://mlsanity.com/paper/2604.11720/) (2026)
- [PGID: Progressive Guided Inversion and Denoising for Robust Watermark Detection](https://mlsanity.com/paper/2605.09319/) (2026)
- [DRIFT: Removing Diffusion Watermarks by Deflecting the Generative Trajectory](https://mlsanity.com/paper/2609.08213/) (2026)
- [Persistent Watermarking of Text-to-Image Models](https://mlsanity.com/paper/2609.39024/) (2026)

## Abstract (arXiv)

Watermarking techniques are vital for protecting intellectual property and preventing fraudulent use of media. Most previous watermarking schemes designed for diffusion models embed a secret key in the initial noise. The resulting pattern is often considered hard to remove and forge into unrelated images. In this paper, we propose a black-box adversarial attack without presuming access to the diffusion model weights. Our attack uses only a single watermarked example and is based on a simple observation: there is a many-to-one mapping between images and initial noises. There are regions in the clean image latent space pertaining to each watermark that get mapped to the same initial noise when inverted. Based on this intuition, we propose an adversarial attack to forge the watermark by introducing perturbations to the images such that we can enter the region of watermarked images. We show that we can also apply a similar approach for watermark removal by learning perturbations to exit this region. We report results on multiple watermarking schemes (Tree-Ring, RingID, WIND, and Gaussian Shading) across two diffusion models (SDv1.4 and SDv2.0). Our results demonstrate the effectiveness of the attack and expose vulnerabilities in the watermarking methods, motivating future research on improving them.