SACE: Concept Erasure at the Semantic Singularity in Visual Autoregressive Models
On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment
also in 1 more
Abstract (arXiv)
The rapid progress of visual autoregressive (VAR) models has unlocked a transformative frontier for high-fidelity text-to-image synthesis, while heightening concerns over the safety alignment of generated content. Naive application of existing erasure techniques to VAR models causes catastrophic semantic collapse and visual artifacts, since they are predominantly designed for the homogeneous denoising steps of diffusion models. To address this foundational challenge, we first propose the Semantic Singularity Axiom, which posits that any target semantic concept embedded within a prompt is definitively locked at Scale-0. Then rigorously validate this axiom through our proposed Incremental Semantic Saliency Analysis (ISSA),which also enable the community to transparently inspect the coarse-to-fine semantic injection process. Guided by this insight, we introduce the first scale-aware concept erasure framework (SACE) for VAR models. By strictly confining interventions to the first scale, our approach couples an Entropy-Regularized Erasure Objective to prevent high-entropy sampling degeneration, alongside a restorative preservation loss to safely anchor the integrity of entangled benign priors. Extensive experiments demonstrate that our method achieves surgical concept erasure performance across various domains with minimal training overhead, timely and elegently resolute the critical safety vulnerabilities inherent in emerging VAR architectures. Code is available at: https://github.com/limerenceysy/SACE}{https://github.com/limerenceysy/SACE.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 54 references that are in the map. First posted on arXiv 14 Jun 2026.
Builds on
44 of 54 references.
References · 44
- 2020Denoising Diffusion Probabilistic Modelscited by 20,952
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
- 2014Microsoft COCO: Common Objects in Contextcited by 17,550
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,028
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,564
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,339
- 2022Scalable Diffusion Models with Transformerscited by 5,973
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,237
- 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,012
- 2016Pixel Recurrent Neural Networkscited by 815
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 799
- 2023Erasing Concepts from Diffusion Modelscited by 535
32 more
- 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 509
- 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 316
- 2024MACE: Mass Concept Erasure in Diffusion Modelscited by 303
- 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 301
- 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 300
- 2023To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Nowcited by 210
- 2023Selective Amnesia: A Continual Learning Approach to Forgetting in Deep Generative Modelscited by 177
- 2023One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applicationscited by 169
- 2024Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Modelscited by 168
- 2023Editing Implicit Assumptions in Text-to-Image Diffusion Modelscited by 129
- 2023Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Eraserscited by 120
- 2024Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradientcited by 69
- 2024SafeGen: Mitigating Sexually Explicit Content Generation in Text-to-Image Modelscited by 65
- 2024R.A.C.E.: Robust Adversarial Concept Erasure for Secure Text-to-Image Diffusion Modelcited by 58
- 2024ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruningcited by 54
- 2024Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservationcited by 50
- 2025InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generationcited by 43
- 2023Towards Safe Self-Distillation of Internet-Scale Text-to-Image Diffusion Modelscited by 41
- 2023All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Modelscited by 33
- 2023Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusioncited by 27
- 2024Separable Multi-Concept Erasure from Diffusion Modelscited by 27
- 2023Implicit Concept Removal of Diffusion Modelscited by 23
- 2024Pruning for Robust Concept Erasing in Diffusion Modelscited by 22
- 2025Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gatecited by 16
- 2024RealEra: Semantic-level Concept Erasure via Neighbor-Concept Miningcited by 15
- 2025Closing the Safety Gap: Surgical Concept Erasure in Visual Autoregressive Modelscited by 8
- 2025Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Modelscited by 8
- 2024Continuous Concepts Removal in Text-to-image Diffusion Modelscited by 6
- 2025EAR: Erasing Concepts from Unified Autoregressive Modelscited by 4
- 2025UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesiscited by 4
- 2026Conditioned Activation Transport for T2I Safety Steeringcited by 3
- 2026KidsNanny: A Two-Stage Multimodal Content Moderation Pipeline Integrating Visual Classification, Object Detection, OCR, and Contextual Reasoning for Child Safetycited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Erasing Without Collateral Damage: Precise Concept Removal in Diffusion Models
- 2025Erased or Dormant? Rethinking Concept Erasure Through Reversibility
- 2026GRACE: Adaptive Concept Erasure with Geometry-Guided Retention in Diffusion Models
- 2026Localized Concept Erasure in Text-to-Image Diffusion Models via High-Level Representation Misdirection
- 2025Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Better
- 2026CleanVideo: Adaptive Concept Erasure for Text-to-Video Diffusion Models
- 2026CoreUnlearn: Rethinking Concept Unlearning through Disentangled Component-Level Erasure in Text-guided Diffusion Models
- 2025A Comprehensive Survey on Concept Erasure in Text-to-Image Diffusion Models