mlsanityData to 7 Oct 2026

STEEX: Steering Counterfactual Explanations with Semantics

arXiv 2111.09094 ↗First posted 17 Nov 2021cs.CVcited by 40

Not on the map. Why?

Abstract (arXiv)

As deep learning models are increasingly used in safety-critical applications, explainability and trustworthiness become major concerns. For simple images, such as low-resolution face portraits, synthesizing visual counterfactual explanations has recently been proposed as a way to uncover the decision mechanisms of a trained classification model. In this work, we address the problem of producing counterfactual explanations for high-quality images and complex scenes. Leveraging recent semantic-to-image models, we propose a new generative counterfactual explanation framework that produces plausible and sparse modifications which preserve the overall scene structure. Furthermore, we introduce the concept of "region-targeted counterfactual explanations", and a corresponding framework, where users can guide the generation of counterfactuals by specifying a set of semantic regions of the query image the explanation must be about. Extensive experiments are conducted on challenging datasets including high-quality portraits (CelebAMask-HQ) and driving scenes (BDD100k). Code is available at https://github.com/valeoai/STEEX

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 51 of its 61 references that are in the map. First posted on arXiv 17 Nov 2021.

Builds on

51 of 61 references.

References · 51

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  8. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  10. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  11. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  12. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
38 more
  1. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  2. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  3. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  4. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  5. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  6. 2017Deep Image Priorcited by 1,097
  7. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,086
  8. 2017VGGFace2: A dataset for recognising faces across pose and agecited by 911
  9. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 731
  10. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  11. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  12. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  13. 2017Compressed Sensing using Generative Modelscited by 337
  14. 2019Counterfactual Visual Explanationscited by 312
  15. 2019SEAN: Image Synthesis with Semantic Region-Adaptive Normalizationcited by 253
  16. 2020Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Surveycited by 252
  17. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 249
  18. 2017Distilling a Neural Network Into a Soft Decision Treecited by 230
  19. 2014Object Detectors Emerge in Deep Scene CNNscited by 178
  20. 2018Isolating Sources of Disentanglement in Variational Autoencoderscited by 167
  21. 2017Interpretable Convolutional Neural Networkscited by 161
  22. 2018Explaining Image Classifiers by Counterfactual Generationcited by 144
  23. 2020You Only Need Adversarial Supervision for Semantic Image Synthesiscited by 129
  24. 2021Explaining in Style: Training a GAN to explain a classifier in StyleSpacecited by 122
  25. 2018Grounding Visual Explanationscited by 92
  26. 2020Explainable Object-induced Action Decision for Autonomous Vehiclescited by 88
  27. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  28. 2019Explanation by Progressive Exaggerationcited by 76
  29. 2020There and Back Again: Revisiting Backpropagation Saliency Methodscited by 60
  30. 2021Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanationscited by 56
  31. 2020SCOUT: Self-aware Discriminant Counterfactual Explanationscited by 52
  32. 2019Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networkscited by 49
  33. 2021Discover the Unknown Biased Attribute of an Image Classifiercited by 46
  34. 2020Semantics and explanation: why counterfactual explanations produce adversarial examples in deep neural networkscited by 24
  35. 2018Causal Learning and Explanation of Deep Neural Networks via Autoencoded Activationscited by 23
  36. 2020High-Fidelity Synthesis with Disentangled Representationcited by 23
  37. 2020To Explain or Not to Explain: A Study on the Necessity of Explanations for Autonomous Vehiclescited by 21
  38. 2020Flexible and Context-Specific AI Explainability: A Multidisciplinary Approachcited by 13

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 12; 2024: 12; 2025: 4; 2026: 5.

40 papers.

Sort:

By area:

  1. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  2. 2022Diffusion Models for Counterfactual Explanationscited by 79
  3. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directionscited by 73
  4. 2023Adversarial Counterfactual Visual Explanationscited by 47
  5. 2023Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generationcited by 41
  6. 2022Making Heads or Tails: Towards Semantically Consistent Visual Counterfactualscited by 30
  7. 2022Behavioral Intention Prediction in Driving Scenes: A Surveycited by 30
  8. 2022OCTET: Object-aware Counterfactual Explanationscited by 28
  9. 2023Text-to-Image Models for Counterfactual Explanations: a Black-Box Approachcited by 28
  10. 2023Latent Diffusion Counterfactual Explanationscited by 17
  11. 2023Explaining Autonomous Driving Actions with Visual Question Answeringcited by 16
  12. 2024Global Counterfactual Directionscited by 15
28 more
  1. 2024Rethinking Visual Counterfactual Explanations Through Region Constraintcited by 14
  2. 2024Explainable AI (XAI) in Image Segmentation in Medicine, Industry, and Beyond: A Surveycited by 11
  3. 2023Deep Backtracking Counterfactuals for Causally Compliant Explanationscited by 8
  4. 2024SAFE-RL: Saliency-Aware Counterfactual Explainer for Deep Reinforcement Learning Policiescited by 8
  5. 2024Safety Implications of Explainable Artificial Intelligence in End-to-End Autonomous Drivingcited by 7
  6. 2024Exploring the Causality of End-to-End Autonomous Drivingcited by 7
  7. 2022On the explainable properties of 1-Lipschitz Neural Networks: An Optimal Transport Perspectivecited by 6
  8. 2024Accurate Explanation Model for Image Classifiers using Class Association Embeddingcited by 6
  9. 2023Tackling Interpretability in Audio Classification Networks with Non-negative Matrix Factorizationcited by 5
  10. 2023SAFE: Saliency-Aware Counterfactual Explanations for DNN-based Automated Driving Systemscited by 5
  11. 2024What could go wrong? Discovering and describing failure modes in computer visioncited by 5
  12. 2024The Gaussian Discriminant Variational Autoencoder (GdVAE): A Self-Explainable Model with Counterfactual Explanationscited by 5
  13. 2025Diffusion Counterfactuals for Image Regressorscited by 5
  14. 2023Interpretable Knowledge Tracing via Response Influence-based Counterfactual Reasoningcited by 4
  15. 2023Counterfactual Explainer Framework for Deep Reinforcement Learning Models Using Policy Distillationcited by 3
  16. 2023Gated Driver Attention Predictorcited by 2
  17. 2024VALD-MD: Visual Attribution via Latent Diffusion for Medical Diagnosticscited by 2
  18. 2024Good Data Is All Imitation Learning Needscited by 2
  19. 2026Auditing Sybil: Explaining Deep Lung Cancer Risk Prediction Through Generative Interventional Attributionscited by 2
  20. 2026Benchmarking Semantic Segmentation Models via Appearance and Geometry Attribute Editingcited by 2
  21. 2024MiMICRI: Towards Domain-centered Counterfactual Explanations of Cardiovascular Image Classification Modelscited by 1
  22. 2025Counterfactual Visual Explanation via Causally-Guided Adversarial Steeringcited by 1
  23. 2022Exploring Effectiveness of Explanations for Appropriate Trust: Lessons from Cognitive Psychologycited by 0
  24. 2025Explaining Low Perception Model Competency with High-Competency Counterfactualscited by 0
  25. 2025Uncertainty-Aware Subset Selection for Robust Visual Explainability under Distribution Shiftscited by 0
  26. 2026Concept-based Visual Counterfactual Explanations with Diffusion Modelscited by 0
  27. 2026FiRe: Fixed-Noise Refinement for Visual Counterfactual Explanationscited by 0
  28. 2026Beyond the Leaderboard: Counterfactual Diagnosis of End-to-End and VLA Driving Policies Under Domain Shiftcited by 0