Towards Dataset Copyright Evasion Attack against Personalized Text-to-Image Diffusion Models
On the map: Diffusion models → Robustness, privacy and adversarial aspects
also in 1 more
Abstract (arXiv)
Text-to-image (T2I) diffusion models enable high-quality image generation conditioned on textual prompts. However, fine-tuning these pre-trained models for personalization raises concerns about unauthorized dataset usage. To address this issue, dataset ownership verification (DOV) has recently been proposed, which embeds watermarks into fine-tuning datasets via backdoor techniques. These watermarks remain dormant on benign samples but produce owner-specified outputs when triggered. Despite its promise, the robustness of DOV against copyright evasion attacks (CEA) remains unexplored. In this paper, we investigate how adversaries can circumvent these mechanisms, enabling models trained on watermarked datasets to bypass ownership verification. We begin by analyzing the limitations of potential attacks achieved by backdoor removal, including TPD and T2IShield. In practice, TPD suffers from inconsistent effectiveness due to randomness, while T2IShield fails when watermarks are embedded as local image patches. To this end, we introduce CEAT2I, the first CEA specifically targeting DOV in T2I diffusion models. CEAT2I consists of three stages: (1) motivated by the observation that T2I models converge faster on watermarked samples with respect to intermediate features rather than training loss, we reliably detect watermarked samples; (2) we iteratively ablate tokens from the prompts of detected samples and monitor feature shifts to identify trigger tokens; and (3) we apply a closed-form concept erasure method to remove the injected watermarks. Extensive experiments demonstrate that CEAT2I effectively evades state-of-the-art DOV mechanisms while preserving model performance. The code is available at https://github.com/csyufei/CEAT2I.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 61 of its 86 references that are in the map. First posted on arXiv 5 May 2025.
Builds on
61 of 86 references.
Closest ancestors · 8
Same direction, newest first
- 2025Retrievals Can Be Detrimental: Unveiling the Backdoor Vulnerability of Retrieval-Augmented Diffusion Models23 Jan
- 2024Towards Reliable Verification of Unauthorized Data Usage in Personalized Text-to-Image Diffusion Models14 Oct
- 2024Defending Text-to-image Diffusion Models: Surprising Efficacy of Textual Perturbations Against Backdoor Attacks28 Aug
- 2024T2IShield: Defending Against Backdoors on Text-to-Image Diffusion Models5 Jul
- 2023Unified Concept Editing in Diffusion Models25 Aug
- 2023VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Models12 Jun
- 2023Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning7 May
- 2023A Recipe for Watermarking Diffusion Models17 Mar
Other references · 53
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
- 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
- 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
- 2016Deep Learning with Differential Privacycited by 2,728
- 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
- 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
- 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
38 more
- 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
- 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
- 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
- 2015Deep Learning and the Information Bottleneck Principlecited by 802
- 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
- 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 642
- 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 324
- 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 282
- 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 263
- 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 208
- 2021Unlearnable Examples: Making Personal Data Unexploitablecited by 193
- 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 182
- 2022Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protectioncited by 86
- 2022Black-box Dataset Ownership Verification via Backdoor Watermarkingcited by 65
- 2022The ArtBench Dataset: Benchmarking Generative Models with Artworkscited by 64
- 2022Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesiscited by 59
- 2023Domain Watermark: Effective and Harmless Dataset Copyright Protection is Closed at Handcited by 57
- 2023Localizing and Editing Knowledge in Text-to-Image Generative Modelscited by 54
- 2022Transferable Unlearnable Examplescited by 43
- 2023Backdoor Defense via Adaptively Splitting Poisoned Datasetcited by 43
- 2023Did You Train on My Dataset? Towards Public Dataset Protection with Clean-Label Backdoor Watermarkingcited by 37
- 2023BAGM: A Backdoor Attack for Manipulating Text-to-Image Generative Modelscited by 37
- 2022One Parameter Defense -- Defending against Data Inference Attacks via Differential Privacycited by 27
- 2024PointNCBW: Towards Dataset Ownership Verification for Point Clouds via Negative Clean-label Backdoor Watermarkcited by 20
- 2024On the Feature Learning in Diffusion Modelscited by 20
- 2020VFL: A Verifiable Federated Learning with Privacy-Preserving for Big Data in Industrial IoTcited by 18
- 2023Unlearnable Examples Give a False Sense of Security: Piercing through Unexploitable Data with Learnable Examplescited by 17
- 2024On the Multi-modal Vulnerability of Diffusion Modelscited by 15
- 2024Towards Faster Training of Diffusion Models: An Inspiration of A Consistency Phenomenoncited by 14
- 2016Individual Differential Privacy: A Utility-Preserving Formulation of Differential Privacy Guaranteescited by 10
- 2024Data Taggants: Dataset Ownership Verification via Harmless Targeted Data Poisoningcited by 8
- 2024Video Watermarking: Safeguarding Your Video from (Unauthorized) Annotations by Video-based LLMscited by 7
- 2024Intellectual Property Protection for Deep Learning Model and Dataset Intelligencecited by 6
- 2025Protecting Your Video Content: Disrupting Automated Video-based LLM Annotationscited by 5
- 2017Halftone Image Watermarking by Content Aware Double-sided Embedding Error Diffusioncited by 4
- 2024Reconstruction of Differentially Private Text Sanitization via Large Language Modelscited by 4
- 2021RBNN: Memory-Efficient Reconfigurable Deep Binary Neural Network with IP Protection for Internet of Thingscited by 3
- 2025CBW: Towards Dataset Ownership Verification for Speaker Verification via Clustering-based Backdoor Watermarkingcited by 3
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Forging the Unforgeable: On the Feasibility of Counterfeit Watermarks in Backdoor-Based Dataset Ownership Verification
- 2024SleeperMark: Towards Robust Watermark against Fine-Tuning Text-to-image Diffusion Models
- 2024Detecting Dataset Abuse in Fine-Tuning Stable Diffusion Models for Text-to-Image Synthesis
- 2024CopyrightMeter: Revisiting Copyright Protection in Text-to-image Models
- 2024Exploiting Watermark-Based Defense Mechanisms in Text-to-Image Diffusion Models for Unauthorized Data Usage
- 2024EnTruth: Enhancing the Traceability of Unauthorized Dataset Usage in Text-to-image Diffusion Models with Minimal and Robust Alterations
- 2024The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright Breaches Without Adjusting Finetuning Pipeline
- 2025Harnessing Frequency Spectrum Insights for Image Copyright Protection Against Diffusion Models