# Explaining in Diffusion: Explaining a Classifier Through Hierarchical Semantics with Text-to-Image Diffusion Models

arXiv: https://arxiv.org/abs/2412.18604 · HTML: https://mlsanity.com/paper/2412.18604/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 17 of its 71 references that are in the map.
- First posted on arXiv 24 Dec 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Selective Annotation Makes Language Models Better Few-Shot Learners](https://mlsanity.com/paper/2209.01975/) (2022)
- [An open access repository of images on plant health to enable the development of mobile disease diagnostics](https://mlsanity.com/paper/1511.08060/) (2015)
- [Explaining Classifiers with Causal Concept Effect (CaCE)](https://mlsanity.com/paper/1907.07165/) (2019)
- [Counterfactual Generative Networks](https://mlsanity.com/paper/2101.06046/) (2021)
- [Dive into Deep Learning](https://mlsanity.com/paper/2106.11342/) (2021)
- [Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation](https://mlsanity.com/paper/2408.04187/) (2024)
- [Explanation by Progressive Exaggeration](https://mlsanity.com/paper/1911.00483/) (2019)
- [Attribution in Scale and Space](https://mlsanity.com/paper/2004.03383/) (2020)
- [NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models](https://mlsanity.com/paper/2312.05390/) (2023)
- [On the Lack of Robust Interpretability of Neural Text Classifiers](https://mlsanity.com/paper/2106.04631/) (2021)
- [Generating Realistic Counterfactuals for Retinal Fundus and OCT Images using Diffusion Models](https://mlsanity.com/paper/2311.11629/) (2023)
- [Explaining the Black-box Smoothly- A Counterfactual Approach](https://mlsanity.com/paper/2101.04230/) (2021)

## Built on by (most cited first, 3 of 3)

- [LoRAShop: Training-Free Multi-Concept Image Generation and Editing with Rectified Flow Transformers](https://mlsanity.com/paper/2505.23758/) (2025)
- [Towards Desiderata-Driven Design of Visual Counterfactual Explainers](https://mlsanity.com/paper/2506.14698/) (2025)
- [Auditing Sybil: Explaining Deep Lung Cancer Risk Prediction Through Generative Interventional Attributions](https://mlsanity.com/paper/2602.02560/) (2026)

## Abstract (arXiv)

Classifiers are important components in many computer vision tasks, serving as the foundational backbone of a wide variety of models employed across diverse applications. However, understanding the decision-making process of classifiers remains a significant challenge. We propose DiffEx, a novel method that leverages the capabilities of text-to-image diffusion models to explain classifier decisions. Unlike traditional GAN-based explainability models, which are limited to simple, single-concept analyses and typically require training a new model for each classifier, our approach can explain classifiers that focus on single concepts (such as faces or animals) as well as those that handle complex scenes involving multiple concepts. DiffEx employs vision-language models to create a hierarchical list of semantics, allowing users to identify not only the overarching semantic influences on classifiers (e.g., the 'beard' semantic in a facial classifier) but also their sub-types, such as 'goatee' or 'Balbo' beard. Our experiments demonstrate that DiffEx is able to cover a significantly broader spectrum of semantics compared to its GAN counterparts, providing a hierarchical tool that delivers a more detailed and fine-grained understanding of classifier decisions.