# AnchorDiff: Training-Free Concept Grounding for MM-DiTs via Anchor-Based Graph Propagation

arXiv: https://arxiv.org/abs/2605.26460 · HTML: https://mlsanity.com/paper/2605.26460/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 26 May 2026.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Seedream 4.0: Toward Next-generation Multimodal Image Generation](https://mlsanity.com/paper/2509.20427/) (2025)
- [What the DAAM: Interpreting Stable Diffusion Using Cross Attention](https://mlsanity.com/paper/2210.04885/) (2022)
- [Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layers](https://mlsanity.com/paper/1604.00825/) (2016)
- [Interpreting CLIP's Image Representation via Text-Based Decomposition](https://mlsanity.com/paper/2310.05916/) (2023)
- [Seedream 3.0 Technical Report](https://mlsanity.com/paper/2504.11346/) (2025)
- [Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter](https://mlsanity.com/paper/2309.02773/) (2023)
- [CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor](https://mlsanity.com/paper/2312.07661/) (2023)
- [ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features](https://mlsanity.com/paper/2502.04320/) (2025)
- [Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models](https://mlsanity.com/paper/2403.14291/) (2024)

## Built on by (most cited first, 1 of 1)

- [Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision](https://mlsanity.com/paper/2609.06491/) (2026)

## Abstract (arXiv)

Multi-Modal Diffusion Transformers (MM-DiTs) encode rich representations for training-free concept grounding, but existing attention-based methods often produce overlapping activations on visually confusable concepts, a failure mode we call concept leakage, where target responses spill over to non-target objects. To address this issue, we propose AnchorDiff, a training-free grounding method that decouples semantic localization from structural refinement. AnchorDiff selects a high-confidence anchor from concept-to-image attention map and propagates it as a one-hot seed over a hybrid graph derived from image-to-image self-attention. The graph uses output-space similarity for dense within-object propagation and a row-wise attention gate to suppress cross-object connections. Additionally, we introduce the Multi-Concept Confusion Dataset, which contains images with multiple visually similar concepts and separate masks, enabling explicit evaluation of concept leakage. Experiments show that AnchorDiff achieves strong grounding performance on ImageNet-Segmentation and PascalVOC, while substantially reducing concept leakage on our Multi-Concept Confusion Dataset.