# Semantic Steering for Controllable Generation: Tuning-Free Concept Erasure in Multimodal Diffusion Transformers

arXiv: https://arxiv.org/abs/2608.12829 · HTML: https://mlsanity.com/paper/2608.12829/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 13 Aug 2026.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 25 of 25)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [MACE: Mass Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2403.06135/) (2024)
- [Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?](https://mlsanity.com/paper/2310.10012/) (2023)
- [MMA-Diffusion: MultiModal Attack on Diffusion Models](https://mlsanity.com/paper/2311.17516/) (2023)
- [Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts](https://mlsanity.com/paper/2309.06135/) (2023)
- [SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation](https://mlsanity.com/paper/2410.12761/) (2024)
- [SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders](https://mlsanity.com/paper/2501.18052/) (2025)
- [Controlling Language and Diffusion Models by Transporting Activations](https://mlsanity.com/paper/2410.23054/) (2024)
- [Set You Straight: Auto-Steering Denoising Trajectories to Sidestep Unwanted Concepts](https://mlsanity.com/paper/2504.12782/) (2025)
- [Open Problems in Machine Unlearning for AI Safety](https://mlsanity.com/paper/2501.04952/) (2025)
- [Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservation](https://mlsanity.com/paper/2410.15618/) (2024)
- [Self-Detoxifying Language Models via Toxification Reversal](https://mlsanity.com/paper/2310.09573/) (2023)
- [Erasing Concepts from Text-to-Image Diffusion Models with Few-shot Unlearning](https://mlsanity.com/paper/2405.07288/) (2024)
- [TraSCE: Trajectory Steering for Concept Erasure](https://mlsanity.com/paper/2412.07658/) (2024)
- [ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment](https://mlsanity.com/paper/2602.17560/) (2026)
- [CASteer: Cross-Attention Steering for Controllable Concept Erasure](https://mlsanity.com/paper/2503.09630/) (2025)

## Built on by (most cited first, 1 of 1)

- [AcFlow: Controlling Text-to-Image Diffusion Transformers via Learned Conditional Activation Flow](https://mlsanity.com/paper/2609.10723/) (2026)

## Abstract (arXiv)

Multimodal Diffusion Transformers (MM-DiTs) have demonstrated remarkable text-to-image generation performance, surpassing traditional U-Net-based diffusion models. Nevertheless, their powerful generative capabilities also raise significant safety concerns, as they may generate sensitive or inappropriate content. While existing concept erasure methods aim to mitigate such risks, most require modifying model parameters, which are often architecture-specific and impractical for deployed larger models. Several tuning-free approaches face challenges when applied to advanced large-scale MM-DiTs due to their deeply embedded knowledge, broad semantic space, and context-dependent text encoders. To address these challenges, we propose to erase concepts by directly manipulating the model's internal representations. Our key insight, derived from an in-depth analysis of MM-DiT's block-wise generative roles, is that text-conditioned semantic representations are most salient in the middle blocks of MM-DiTs. Based on this, we extract representations of an unwanted concept and a desirable safe one from the middle block, construct a steering vector from their difference, and inject this single vector into consecutive early and middle blocks. By operating exclusively on the sparse text-branch tokens and leveraging the straight sampling trajectory of rectified flow, our method achieves effective concept erasure with negligible overhead and without any training. Extensive experiments across MM-DiT models demonstrate that our method achieves state-of-the-art performance in erasing diverse concepts, enables effective control over the final output, and remains robust to adversarial attacks.