# Enabling Preference-driven Unlearning in Few-step Distilled Text-to-Image Diffusion Models

arXiv: https://arxiv.org/abs/2610.10859 · HTML: https://mlsanity.com/paper/2610.10859/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 12 of its 104 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)
- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [Text-to-image Diffusion Models in Generative AI: A Survey](https://mlsanity.com/paper/2303.07909/) (2023)
- [Red-Teaming the Stable Diffusion Safety Filter](https://mlsanity.com/paper/2210.04610/) (2022)
- [Wuerstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models](https://mlsanity.com/paper/2306.00637/) (2023)
- [Continual Unlearning for Foundational Text-to-Image Models without Generalization Erosion](https://mlsanity.com/paper/2503.13769/) (2025)
- [Concept Unlearning by Modeling Key Steps of Diffusion Process](https://mlsanity.com/paper/2507.06526/) (2025)
- [Locality-Aware Continual Unlearning for Diffusion Models](https://mlsanity.com/paper/2512.02657/) (2025)
- [ContinualFlow: Learning and Unlearning with Neural Flow Matching](https://mlsanity.com/paper/2506.18747/) (2025)
- [SafetyPairs: Isolating Safety Critical Image Features with Counterfactual Image Generation](https://mlsanity.com/paper/2510.21120/) (2025)

## Abstract (arXiv)

Text-to-image diffusion models are increasingly distilled into few-step variants and being deployed to enable fast inference. However, their ability to generate harmful or undesired content poses significant safety risks. Data-driven unlearning methods suppress targeted generations by fine-tuning model weights using specialized unlearning objectives. Crucially, these objectives implicitly rely on multi-step denoising dynamics, an assumption that breaks down for few-step distilled (FSD) models, resulting in ineffective forgetting. Furthermore, performing unlearning on the non-distilled base model and subsequently re-distilling it to obtain an unlearned FSD model incurs substantial computational and time overhead, making it impractical in many settings. Hence, we address this limitation with a preference-driven unlearning framework that revisits Direct Preference Optimization (DPO) for diffusion models. We show that standard DPO and its unlearning derivatives, formulated around noise-prediction error, transfer poorly to FSD models due to their altered generation dynamics. To overcome this, we introduce a modified preference optimization formulation explicitly aligned with the few-step generation properties, enabling direct concept removal in FSD models while preserving few-step efficiency and maintaining strong retention of desirable (non-targeted) capabilities. We evaluate our framework primarily on identity and NSFW (nudity) removal tasks and also extend our method to object-level unlearning. Extensive experiments demonstrate consistent and effective forgetting, and strong retention performance, establishing our method as a practical and principled solution for unlearning in FSD models.