# Anti-Reference: Universal and Immediate Defense Against Reference-Based Generation

arXiv: https://arxiv.org/abs/2412.05980 · HTML: https://mlsanity.com/paper/2412.05980/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- First posted on arXiv 8 Dec 2024.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Built on by (most cited first, 18 of 18)

- [EasyControl: Adding Efficient and Flexible Control for Diffusion Transformer](https://mlsanity.com/paper/2503.07027/) (2025)
- [Any2AnyTryon: Leveraging Adaptive Position Embeddings for Versatile Virtual Clothing Tasks](https://mlsanity.com/paper/2501.15891/) (2025)
- [Dual Data Alignment Makes AI-Generated Image Detector Easier Generalizable](https://mlsanity.com/paper/2505.14359/) (2025)
- [EEdit: Rethinking the Spatial and Temporal Redundancy for Efficient Image Editing](https://mlsanity.com/paper/2503.10270/) (2025)
- [LayerTracer: Cognitive-Aligned Layered SVG Synthesis via Diffusion Transformer](https://mlsanity.com/paper/2502.01105/) (2025)
- [ContextFlow: Training-Free Video Object Editing via Adaptive Context Enrichment](https://mlsanity.com/paper/2509.17818/) (2025)
- [RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformers](https://mlsanity.com/paper/2506.02528/) (2025)
- [PhotoDoodle: Learning Artistic Image Editing from Few-Shot Pairwise Data](https://mlsanity.com/paper/2502.14397/) (2025)
- [IDProtector: An Adversarial Noise Encoder to Protect Against ID-Preserving Image Generation](https://mlsanity.com/paper/2412.11638/) (2024)
- [DiffSim: Taming Diffusion Models for Evaluating Visual Similarity](https://mlsanity.com/paper/2412.14580/) (2024)
- [MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness Against VLM-based Attacks](https://mlsanity.com/paper/2506.05982/) (2025)
- [DiffSeg30k: A Multi-Turn Diffusion Editing Benchmark for Localized AIGC Detection](https://mlsanity.com/paper/2511.19111/) (2025)
- [MultiMotion: Multi Subject Video Motion Transfer via Video Diffusion Transformer](https://mlsanity.com/paper/2512.07500/) (2025)
- [Unified Long Video Inpainting and Outpainting via Overlapping High-Order Co-Denoising](https://mlsanity.com/paper/2511.03272/) (2025)
- [TokenPure: Watermark Removal through Tokenized Appearance and Structural Guidance](https://mlsanity.com/paper/2512.01314/) (2025)
- [AutoMIA: Improved Baselines for Membership Inference Attack via Agentic Self-Exploration](https://mlsanity.com/paper/2604.01014/) (2026)
- [A Survey on Adversarial Attacks and Defenses for Diffusion Models Across Multiple Modalities](https://mlsanity.com/paper/2609.05503/) (2026)
- [No Way To Steal My Face: Proactive Defense Against Identity-Preserving Personalized Generation](https://mlsanity.com/paper/2610.19717/) (2026)

## Abstract (arXiv)

Diffusion models have revolutionized generative modeling with their exceptional ability to produce high-fidelity images. However, misuse of such potent tools can lead to the creation of fake news or disturbing content targeting individuals, resulting in significant social harm. In this paper, we introduce Anti-Reference, a novel method that protects images from the threats posed by reference-based generation techniques by adding imperceptible adversarial noise to the images. We propose a unified loss function that enables joint attacks on fine-tuning-based customization methods, non-fine-tuning customization methods, and human-centric driving methods. Based on this loss, we train a Adversarial Noise Encoder to predict the noise or directly optimize the noise using the PGD method. Our method shows certain transfer attack capabilities, effectively challenging both gray-box models and some commercial APIs. Extensive experiments validate the performance of Anti-Reference, establishing a new benchmark in image security.