# UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Models

arXiv: https://arxiv.org/abs/2402.11846 · HTML: https://mlsanity.com/paper/2402.11846/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 71 of its 102 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 30 of 71)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Certified Data Removal from Machine Learning Models](https://mlsanity.com/paper/1911.03030/) (2019)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [Object Detection in 20 Years: A Survey](https://mlsanity.com/paper/1905.05055/) (2019)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Scaling up GANs for Text-to-Image Synthesis](https://mlsanity.com/paper/2303.05511/) (2023)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Making AI Forget You: Data Deletion in Machine Learning](https://mlsanity.com/paper/1907.05012/) (2019)
- [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](https://mlsanity.com/paper/2305.14720/) (2023)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [InstantID: Zero-shot Identity-Preserving Generation in Seconds](https://mlsanity.com/paper/2401.07519/) (2024)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation](https://mlsanity.com/paper/2310.12508/) (2023)
- [Remember What You Want to Forget: Algorithms for Machine Unlearning](https://mlsanity.com/paper/2103.03279/) (2021)
- [Amnesiac Machine Learning](https://mlsanity.com/paper/2010.10981/) (2020)
- [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](https://mlsanity.com/paper/2212.02499/) (2022)
- [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023)

## Built on by (most cited first, 30 of 50)

- [Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning](https://mlsanity.com/paper/2410.07163/) (2024)
- [SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders](https://mlsanity.com/paper/2501.18052/) (2025)
- [CLEAR: Character Unlearning in Textual and Visual Modalities](https://mlsanity.com/paper/2410.18057/) (2024)
- [Machine Unlearning: A Comprehensive Survey](https://mlsanity.com/paper/2405.07406/) (2024)
- [Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for Text-to-Image Diffusion Model Unlearning](https://mlsanity.com/paper/2410.05664/) (2024)
- [Mass Concept Erasure in Diffusion Models with Concept Hierarchy](https://mlsanity.com/paper/2601.03305/) (2026)
- [Replication in Visual Diffusion Models: A Survey and Outlook](https://mlsanity.com/paper/2408.00001/) (2024)
- [Erasing Concepts, Steering Generations: A Comprehensive Survey of Concept Suppression](https://mlsanity.com/paper/2505.19398/) (2025)
- [SAEmnesia: Erasing Concepts in Diffusion Models with Supervised Sparse Autoencoders](https://mlsanity.com/paper/2509.21379/) (2025)
- [Continual Unlearning for Text-to-Image Diffusion Models: A Regularization Perspective](https://mlsanity.com/paper/2511.07970/) (2025)
- [Deep Unlearn: Benchmarking Machine Unlearning for Image Classification](https://mlsanity.com/paper/2410.01276/) (2024)
- [CE-SDWV: Effective and Efficient Concept Erasure for Text-to-Image Diffusion Models via a Semantic-Driven Word Vocabulary](https://mlsanity.com/paper/2501.15562/) (2025)
- [Sparse Autoencoder as a Zero-Shot Classifier for Concept Erasing in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2503.09446/) (2025)
- [AdvAnchor: Enhancing Diffusion Model Unlearning with Adversarial Anchors](https://mlsanity.com/paper/2501.00054/) (2024)
- [Concept Unlearning by Modeling Key Steps of Diffusion Process](https://mlsanity.com/paper/2507.06526/) (2025)
- [Reference-Specific Unlearning Metrics Can Hide the Truth: A Reality Check](https://mlsanity.com/paper/2510.12981/) (2025)
- [Locality-Aware Continual Unlearning for Diffusion Models](https://mlsanity.com/paper/2512.02657/) (2025)
- [Generative Model Unlearning: A Survey through Target Events, Unlearning Operators, and Evaluation Protocols](https://mlsanity.com/paper/2507.19894/) (2025)
- [Low-Rank Continual Personalization of Diffusion Models](https://mlsanity.com/paper/2410.04891/) (2024)
- [Localized Concept Erasure in Text-to-Image Diffusion Models via High-Level Representation Misdirection](https://mlsanity.com/paper/2602.19631/) (2026)
- [SPQR: A Multi-Dimensional Benchmark for Safety Alignment under Benign Model Adaptation](https://mlsanity.com/paper/2511.19558/) (2025)
- [SPARE: Self-distillation for PARameter-Efficient Removal](https://mlsanity.com/paper/2602.07058/) (2026)
- [Selective Fine-Tuning for Targeted and Robust Concept Unlearning](https://mlsanity.com/paper/2602.07919/) (2026)
- [Disentangled Sparse Representations for Concept-Separated Diffusion Unlearning](https://mlsanity.com/paper/2605.12122/) (2026)
- [Comprehensive Evaluation and Analysis for NSFW Concept Erasure in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2505.15450/) (2025)
- [LoRAShield: Data-Free Editing Alignment for Secure Personalized LoRA Sharing](https://mlsanity.com/paper/2507.07056/) (2025)
- [Automating Evaluation of Diffusion Model Unlearning with (Vision-) Language Model World Knowledge](https://mlsanity.com/paper/2507.07137/) (2025)
- [Unlearning in Diffusion models under Data Constraints: A Variational Inference Approach](https://mlsanity.com/paper/2510.04058/) (2025)
- [SUGAR: A Sweeter Spot for Generative Unlearning of Many Identities](https://mlsanity.com/paper/2512.06562/) (2025)
- [Look But Don't Touch with Sparse Autoencoders for Unlearning in Diffusion Models](https://mlsanity.com/paper/2606.31699/) (2026)

## Abstract (arXiv)

The technological advancements in diffusion models (DMs) have demonstrated unprecedented capabilities in text-to-image generation and are widely used in diverse applications. However, they have also raised significant societal concerns, such as the generation of harmful content and copyright disputes. Machine unlearning (MU) has emerged as a promising solution, capable of removing undesired generative capabilities from DMs. However, existing MU evaluation systems present several key challenges that can result in incomplete and inaccurate assessments. To address these issues, we propose UnlearnCanvas, a comprehensive high-resolution stylized image dataset that facilitates the evaluation of the unlearning of artistic styles and associated objects. This dataset enables the establishment of a standardized, automated evaluation framework with 7 quantitative metrics assessing various aspects of the unlearning performance for DMs. Through extensive experiments, we benchmark 9 state-of-the-art MU methods for DMs, revealing novel insights into their strengths, weaknesses, and underlying mechanisms. Additionally, we explore challenging unlearning scenarios for DMs to evaluate worst-case performance against adversarial prompts, the unlearning of finer-scale concepts, and sequential unlearning. We hope that this study can pave the way for developing more effective, accurate, and robust DM unlearning methods, ensuring safer and more ethical applications of DMs in the future. The dataset, benchmark, and codes are publicly available at https://unlearn-canvas.netlify.app/.