# Rethinking Visual Counterfactual Explanations Through Region Constraint

arXiv: https://arxiv.org/abs/2410.12591 · HTML: https://mlsanity.com/paper/2410.12591/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 73 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Builds on (most cited first, 30 of 60)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)

## Built on by (most cited first, 14 of 14)

- [Causally Steered Diffusion for Automated Video Counterfactual Generation](https://mlsanity.com/paper/2506.14404/) (2025)
- [System-Embedded Diffusion Bridge Models](https://mlsanity.com/paper/2506.23726/) (2025)
- [V-CECE: Visual Counterfactual Explanations via Conceptual Edits](https://mlsanity.com/paper/2509.16567/) (2025)
- [Model Science: getting serious about verification, explanation and control of AI systems](https://mlsanity.com/paper/2508.20040/) (2025)
- [Photorealistic Inpainting for Perturbation-based Explanations in Ecological Monitoring](https://mlsanity.com/paper/2510.03317/) (2025)
- [Counterfactual Explanations on Robust Perceptual Geodesics](https://mlsanity.com/paper/2601.18678/) (2026)
- [Auditing Sybil: Explaining Deep Lung Cancer Risk Prediction Through Generative Interventional Attributions](https://mlsanity.com/paper/2602.02560/) (2026)
- [MaskDiME: Adaptive Masked Diffusion for Precise and Efficient Visual Counterfactual Explanations](https://mlsanity.com/paper/2602.18792/) (2026)
- [Back to the Feature: Explaining Video Classifiers with Video Counterfactual Explanations](https://mlsanity.com/paper/2511.20295/) (2025)
- [LINE: LLM-based Iterative Neuron Explanations for Vision Models](https://mlsanity.com/paper/2604.08039/) (2026)
- [U-CECE: A Universal Multi-Resolution Framework for Conceptual Counterfactual Explanations](https://mlsanity.com/paper/2604.08295/) (2026)
- [FiRe: Fixed-Noise Refinement for Visual Counterfactual Explanations](https://mlsanity.com/paper/2608.08664/) (2026)
- [MedDiME: Efficient Latent Diffusion with Adaptive Masking for Medical Counterfactual Generation](https://mlsanity.com/paper/2609.15647/) (2026)
- [Targeted Visual Counterfactual Explanations for Contrastive Vision-Language Model](https://mlsanity.com/paper/2609.37638/) (2026)

## Abstract (arXiv)

Visual counterfactual explanations (VCEs) have recently gained immense popularity as a tool for clarifying the decision-making process of image classifiers. This trend is largely motivated by what these explanations promise to deliver -- indicate semantically meaningful factors that change the classifier's decision. However, we argue that current state-of-the-art approaches lack a crucial component -- the region constraint -- whose absence prevents from drawing explicit conclusions, and may even lead to faulty reasoning due to phenomenons like confirmation bias. To address the issue of previous methods, which modify images in a very entangled and widely dispersed manner, we propose region-constrained VCEs (RVCEs), which assume that only a predefined image region can be modified to influence the model's prediction. To effectively sample from this subclass of VCEs, we propose Region-Constrained Counterfactual Schr"odinger Bridges (RCSB), an adaptation of a tractable subclass of Schr"odinger Bridges to the problem of conditional inpainting, where the conditioning signal originates from the classifier of interest. In addition to setting a new state-of-the-art by a large margin, we extend RCSB to allow for exact counterfactual reasoning, where the predefined region contains only the factor of interest, and incorporating the user to actively interact with the RVCE by predefining the regions manually.