# AquaLoRA: Toward White-box Protection for Customized Stable Diffusion Models via Watermark LoRA

arXiv: https://arxiv.org/abs/2405.11135 · HTML: https://mlsanity.com/paper/2405.11135/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 18 May 2024.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)
- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models](https://mlsanity.com/paper/2302.04867/) (2023)
- [HiDDeN: Hiding Data With Deep Networks](https://mlsanity.com/paper/1807.09937/) (2018)
- [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://mlsanity.com/paper/2306.09344/) (2023)
- [The Stable Signature: Rooting Watermarks in Latent Diffusion Models](https://mlsanity.com/paper/2303.15435/) (2023)
- [LCM-LoRA: A Universal Stable-Diffusion Acceleration Module](https://mlsanity.com/paper/2311.05556/) (2023)
- [Artificial Fingerprinting for Generative Models: Rooting Deepfake Attribution in Training Data](https://mlsanity.com/paper/2007.08457/) (2020)
- [Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robust](https://mlsanity.com/paper/2305.20030/) (2023)
- [Robust Invisible Video Watermarking with Attention](https://mlsanity.com/paper/1909.01285/) (2019)
- [A Recipe for Watermarking Diffusion Models](https://mlsanity.com/paper/2303.10137/) (2023)
- [Invisible Image Watermarks Are Provably Removable Using Generative AI](https://mlsanity.com/paper/2306.01953/) (2023)
- [RoSteALS: Robust Steganography using Autoencoder Latent Space](https://mlsanity.com/paper/2304.03400/) (2023)

## Built on by (most cited first, 30 of 46)

- [Watermark Anything with Localized Messages](https://mlsanity.com/paper/2411.07231/) (2024)
- [Black-Box Forgery Attacks on Semantic Watermarks for Diffusion Models](https://mlsanity.com/paper/2412.03283/) (2024)
- [WMAdapter: Adding WaterMark Control to Latent Diffusion Models](https://mlsanity.com/paper/2406.08337/) (2024)
- [SleeperMark: Towards Robust Watermark against Fine-Tuning Text-to-image Diffusion Models](https://mlsanity.com/paper/2412.04852/) (2024)
- [GaussMarker: Robust Dual-Domain Watermark for Diffusion Models](https://mlsanity.com/paper/2506.11444/) (2025)
- [Semantic Watermarking Reinvented: Enhancing Robustness and Generation Quality with Fourier Integrity](https://mlsanity.com/paper/2509.07647/) (2025)
- [FoldMark: Protecting Protein Generative Models with Watermarking](https://mlsanity.com/paper/2410.20354/) (2024)
- [MaXsive: High-Capacity and Robust Training-Free Generative Image Watermarking in Diffusion Models](https://mlsanity.com/paper/2507.21195/) (2025)
- [Secure and Robust Watermarking for AI-generated Images: A Comprehensive Survey](https://mlsanity.com/paper/2510.02384/) (2025)
- [Secure and Efficient Watermarking for Latent Diffusion Models in Model Distribution Scenarios](https://mlsanity.com/paper/2502.13345/) (2025)
- [GaussianSeal: Rooting Adaptive Watermarks for 3D Gaussian Generation Model](https://mlsanity.com/paper/2503.00531/) (2025)
- [ImageSentinel: Protecting Visual Datasets from Unauthorized Retrieval-Augmented Image Generation](https://mlsanity.com/paper/2510.12119/) (2025)
- [Watermarking Visual Concepts for Diffusion Models](https://mlsanity.com/paper/2411.11688/) (2024)
- [SPDMark: Selective Parameter Displacement for Robust Video Watermarking](https://mlsanity.com/paper/2512.12090/) (2025)
- [Learning to Watermark in the Latent Space of Generative Models](https://mlsanity.com/paper/2601.16140/) (2026)
- [Cert-LAS: Toward Certified Model Ownership Verification for Text-to-Image Diffusion Models via Layer-Adaptive Smoothing](https://mlsanity.com/paper/2605.29809/) (2026)
- [SAT-LDM: Provably Generalizable Image Watermarking for Latent Diffusion Models with Self-Augmented Training](https://mlsanity.com/paper/2501.00463/) (2024)
- [Hiding Images in Diffusion Models by Editing Learned Score Functions](https://mlsanity.com/paper/2503.18459/) (2025)
- [PT-Mark: Invisible Watermarking for Text-to-image Diffusion Models via Semantic-aware Pivotal Tuning](https://mlsanity.com/paper/2504.10853/) (2025)
- [OptMark: Robust Multi-bit Diffusion Watermarking via Inference Time Optimization](https://mlsanity.com/paper/2508.21727/) (2025)
- [OSI: One-step Inversion Excels in Extracting Diffusion Watermarks](https://mlsanity.com/paper/2602.09494/) (2026)
- [Towards Backdoor-Based Ownership Verification for Vision-Language-Action Models](https://mlsanity.com/paper/2605.09005/) (2026)
- [An Efficient Watermarking Method for Latent Diffusion Models via Low-Rank Adaptation and Dynamic Loss Weighting](https://mlsanity.com/paper/2410.20202/) (2024)
- [Spread them Apart: Towards Robust Watermarking of Generated Content](https://mlsanity.com/paper/2502.07845/) (2025)
- [SWA-LDM: Toward Stealthy Watermarks for Latent Diffusion Models](https://mlsanity.com/paper/2502.10495/) (2025)
- [Your Text Encoder Can Be An Object-Level Watermarking Controller](https://mlsanity.com/paper/2503.11945/) (2025)
- [TraceMark-LDM: Authenticatable Watermarking for Latent Diffusion Models via Binary-Guided Rearrangement](https://mlsanity.com/paper/2503.23332/) (2025)
- [SOLIDO: A Robust Watermarking Method for Speech Synthesis via Low-Rank Adaptation](https://mlsanity.com/paper/2504.15035/) (2025)
- [Invariant-based Robust Weights Watermark for Large Language Models](https://mlsanity.com/paper/2507.08288/) (2025)
- [AuthenLoRA: Entangling Stylization with Imperceptible Watermarks for Copyright-Secure LoRA Adapters](https://mlsanity.com/paper/2511.21216/) (2025)

## Abstract (arXiv)

Diffusion models have achieved remarkable success in generating high-quality images. Recently, the open-source models represented by Stable Diffusion (SD) are thriving and are accessible for customization, giving rise to a vibrant community of creators and enthusiasts. However, the widespread availability of customized SD models has led to copyright concerns, like unauthorized model distribution and unconsented commercial use. To address it, recent works aim to let SD models output watermarked content for post-hoc forensics. Unfortunately, none of them can achieve the challenging white-box protection, wherein the malicious user can easily remove or replace the watermarking module to fail the subsequent verification. For this, we propose textttmethod as the first implementation under this scenario. Briefly, we merge watermark information into the U-Net of Stable Diffusion Models via a watermark Low-Rank Adaptation (LoRA) module in a two-stage manner. For watermark LoRA module, we devise a scaling matrix to achieve flexible message updates without retraining. To guarantee fidelity, we design Prior Preserving Fine-Tuning (PPFT) to ensure watermark learning with minimal impacts on model distribution, validated by proofs. Finally, we conduct extensive experiments and ablation studies to verify our design.