# MarkPlugger: Generalizable Watermark Framework for Latent Diffusion Models without Retraining

arXiv: https://arxiv.org/abs/2404.05607 · HTML: https://mlsanity.com/paper/2404.05607/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 8 Apr 2024.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)
- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 26 of 26)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [HiDDeN: Hiding Data With Deep Networks](https://mlsanity.com/paper/1807.09937/) (2018)
- [The Stable Signature: Rooting Watermarks in Latent Diffusion Models](https://mlsanity.com/paper/2303.15435/) (2023)
- [StegaStamp: Invisible Hyperlinks in Physical Photographs](https://mlsanity.com/paper/1904.05343/) (2019)
- [UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild](https://mlsanity.com/paper/2305.11147/) (2023)
- [Gaussian Shading: Provable Performance-Lossless Image Watermarking for Diffusion Models](https://mlsanity.com/paper/2404.04956/) (2024)
- [Watermarking Images in Self-Supervised Latent Spaces](https://mlsanity.com/paper/2112.09581/) (2021)
- [Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence](https://mlsanity.com/paper/2209.02970/) (2022)
- [WOUAF: Weight Modulation for User Attribution and Fingerprinting in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2306.04744/) (2023)
- [Attack-Resilient Image Watermarking Using Stable Diffusion](https://mlsanity.com/paper/2401.04247/) (2024)
- [Deepfakes, Misinformation, and Disinformation in the Era of Frontier AI, Generative AI, and Large AI Models](https://mlsanity.com/paper/2311.17394/) (2023)
- [De-END: Decoder-driven Watermarking Network](https://mlsanity.com/paper/2206.13032/) (2022)
- [T2IW: Joint Text to Image & Watermark Generation](https://mlsanity.com/paper/2309.03815/) (2023)
- [Hey That's Mine Imperceptible Watermarks are Preserved in Diffusion Generated Outputs](https://mlsanity.com/paper/2308.11123/) (2023)

## Built on by (most cited first, 5 of 5)

- [Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advances](https://mlsanity.com/paper/2410.18775/) (2024)
- [WMAdapter: Adding WaterMark Control to Latent Diffusion Models](https://mlsanity.com/paper/2406.08337/) (2024)
- [AuthenLoRA: Entangling Stylization with Imperceptible Watermarks for Copyright-Secure LoRA Adapters](https://mlsanity.com/paper/2511.21216/) (2025)
- [Off-The-Shelf Image-to-Image Models Are All You Need To Defeat Image Protection Schemes](https://mlsanity.com/paper/2602.22197/) (2026)
- [Luminark: Training-free, Probabilistically-Certified Watermarking for General Vision Generative Models](https://mlsanity.com/paper/2601.01085/) (2026)

## Abstract (arXiv)

Today, the family of latent diffusion models (LDMs) has gained prominence for its high quality outputs and scalability. This has also raised security concerns on social media, as malicious users can create and disseminate harmful content. Existing approaches typically involve training specific components or entire generative models to embed a watermark in generated images for traceability and responsibility. However, in the fast-evolving era of AI-generated content (AIGC), the rapid iteration and modification of LDMs makes retraining with watermark models costly. To address the problem, we propose MarkPlugger, a generalizable plug-and-play watermark framework without LDM retraining. In particular, to reduce the disturbance of the watermark on the semantics of the generated image, we try to identify a watermark representation that is approaching orthogonal to the semantic in latent space, and apply an additive fusion strategy for the watermark and the semantic. Without modifying any components of the LDMs, we embed diverse watermarks in latent space, adapting to the denoising process. Our experimental findings reveal that our method effectively harmonizes image quality and watermark recovery rate. We also have validated that our method is generalized to multiple official versions and modified variants of LDMs, even without retraining the watermark model. Furthermore, it performs robustly under various attacks of different intensities.