mlsanityData to 7 Oct 2026

Diffexplainer: Towards Cross-modal Global Explanations with Diffusion Models

arXiv 2404.02618 ↗First posted 3 Apr 2024cs.CV cs.AIcited by 5

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

We present DiffExplainer, a novel framework that, leveraging language-vision models, enables multimodal global explainability. DiffExplainer employs diffusion models conditioned on optimized text prompts, synthesizing images that maximize class outputs and hidden features of a classifier, thus providing a visual tool for explaining decisions. Moreover, the analysis of generated visual descriptions allows for automatic identification of biases and spurious features, as opposed to traditional methods that often rely on manual intervention. The cross-modal transferability of language-vision models also enables the possibility to describe decisions in a more human-interpretable way, i.e., through text. We conduct comprehensive experiments, which include an extensive user study, demonstrating the effectiveness of DiffExplainer on 1) the generation of high-quality images explaining model decisions, surpassing existing activation maximization methods, and 2) the automated identification of biases and spurious features.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 34 of its 38 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

34 of 38 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023Segment Anythingcited by 8,158
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  6. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  7. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  10. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  11. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  12. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
22 more
  1. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  2. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  3. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  4. 2020Concept Bottleneck Modelscited by 1,022
  5. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
  6. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
  7. 2015Understanding Neural Networks Through Deep Visualizationcited by 332
  8. 2020Environment Inference for Invariant Learningcited by 306
  9. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 287
  10. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  11. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulecited by 211
  12. 2020No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problemscited by 208
  13. 2016Synthesizing the preferred inputs for neurons in neural networks via deep generator networkscited by 181
  14. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  15. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  16. 2016Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networkscited by 106
  17. 2015Visualizing Deep Convolutional Neural Networks Using Natural Pre-Imagescited by 93
  18. 2020Understanding Failures of Deep Networks via Robust Feature Extractioncited by 56
  19. 2021Meaningfully Debugging Model Mistakes using Conceptual Counterfactual Explanationscited by 53
  20. 2021Unsupervised Learning of Debiased Representations with Pseudo-Attributescited by 44
  21. 2022Missingness Bias in Model Debuggingcited by 36
  22. 2022Freeze then Train: Towards Provable Representation Learning under Spurious Correlations and Feature Noisecited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025DEXTER: Diffusion-Guided EXplanations with TExtual Reasoning for Vision Modelscited by 4
  2. 2026A Distributional View for Visual Mechanistic Interpretability: KL-Minimal Soft-Constraint Principlecited by 1
  3. 2026LEMON: Local Explanations via Modality-aware OptimizatioNcited by 0
  4. 2026UNBOX: Unveiling Black-box visual models with Natural-languagecited by 0
  5. 2026LINE: LLM-based Iterative Neuron Explanations for Vision Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Explaining in Diffusion: Explaining a Classifier Through Hierarchical Semantics with Text-to-Image Diffusion Models
  2. 2024GIFT: A Framework Towards Global Interpretable Faithful Textual Explanations of Vision Classifiers
  3. 2026GridVQA-X: A Framework for Evaluating Multimodal Explainability Methods
  4. 2023The Hidden Language of Diffusion Models
  5. 2026Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability
  6. 2026From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making
  7. 2026Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language Models
  8. 2025GLIMPSE: Holistic Cross-Modal Explainability for Large Vision-Language Models