# EMMA: Concept Erasure Benchmark with Comprehensive Semantic Metrics and Diverse Categories

arXiv: https://arxiv.org/abs/2512.17320 · HTML: https://mlsanity.com/paper/2512.17320/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 63 references that are in the map.
- First posted on arXiv 19 Dec 2025.

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://mlsanity.com/paper/2306.09344/) (2023)
- [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023)
- [MACE: Mass Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2403.06135/) (2024)
- [Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.17591/) (2023)
- [Red-Teaming the Stable Diffusion Safety Filter](https://mlsanity.com/paper/2210.04610/) (2022)
- [Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale](https://mlsanity.com/paper/2211.03759/) (2022)
- [Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?](https://mlsanity.com/paper/2310.10012/) (2023)
- [Stable Bias: Analyzing Societal Representations in Diffusion Models](https://mlsanity.com/paper/2303.11408/) (2023)
- [To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now](https://mlsanity.com/paper/2310.11868/) (2023)
- [Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models](https://mlsanity.com/paper/2305.13873/) (2023)
- [One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications](https://mlsanity.com/paper/2312.16145/) (2023)
- [UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANs](https://mlsanity.com/paper/2311.09257/) (2023)
- [Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?](https://mlsanity.com/paper/2202.06675/) (2022)
- [Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2407.12383/) (2024)
- [Circumventing Concept Erasure Methods For Text-to-Image Generative Models](https://mlsanity.com/paper/2308.01508/) (2023)
- [ML-Decoder: Scalable and Versatile Classification Head](https://mlsanity.com/paper/2111.12933/) (2021)
- [Instruct-Imagen: Image Generation with Multi-modal Instruction](https://mlsanity.com/paper/2401.01952/) (2024)
- [Erasing Undesirable Influence in Diffusion Models](https://mlsanity.com/paper/2401.05779/) (2024)

## Built on by (most cited first, 4 of 4)

- [To Erase, or Not to Erase: Robust Training-Free Concept Erasure with Preservation aware Adaptive Ranked Subspace Expansion](https://mlsanity.com/paper/2607.23492/) (2026)
- [Prototype-Guided Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2603.08271/) (2026)
- [PROBE: Diagnosing Residual Concept Capacity in Erased Text-to-Video Diffusion Models](https://mlsanity.com/paper/2603.21547/) (2026)
- [Mosaic: Compositional Multi-Concept Erasure via Vector Field Blending](https://mlsanity.com/paper/2605.25574/) (2026)

## Abstract (arXiv)

The widespread adoption of text-to-image (T2I) generation has raised concerns about privacy, bias, and copyright violations. Concept erasure techniques offer a promising solution by selectively removing undesired concepts from pre-trained models without requiring full retraining. However, these methods are often evaluated on a limited set of concepts, relying on overly simplistic and direct prompts. To test the boundaries of concept erasure techniques, and assess whether they truly remove targeted concepts from model representations, we introduce EMMA, a benchmark that evaluates five key dimensions of concept erasure over 13 metrics. EMMA goes beyond standard metrics like image quality and time efficiency, testing robustness under challenging conditions, including indirect descriptions, visually similar non-target concepts, and potential gender and ethnicity bias, providing a socially aware analysis of method behavior. Using EMMA, we analyze five concept erasure methods across five domains (objects, celebrities, art styles, NSFW, and copyright). Our results show that existing methods struggle with implicit prompts (i.e., generating the erased concept when it is indirectly referenced) and visually similar non-target concepts (i.e., failing to generate non-target concepts resembling the erased one), while some amplify gender and ethnicity bias compared to the original model. Code and prompts are available at https://github.com/lobsterlulu/EMMA.