# Semantic Prioritization in Visual Counterfactual Explanations with Weighted Segmentation and Auto-Adaptive Region Selection

arXiv: https://arxiv.org/abs/2511.12992 · HTML: https://mlsanity.com/paper/2511.12992/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 17 Nov 2025.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [Full-Gradient Representation for Neural Network Visualization](https://mlsanity.com/paper/1905.00780/) (2019)
- [Explaining Image Classifiers by Counterfactual Generation](https://mlsanity.com/paper/1807.08024/) (2018)
- [Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification](https://mlsanity.com/paper/2108.08728/) (2021)
- [Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop](https://mlsanity.com/paper/2007.11110/) (2020)
- [SCOUT: Self-aware Discriminant Counterfactual Explanations](https://mlsanity.com/paper/2004.07769/) (2020)
- [Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networks](https://mlsanity.com/paper/1904.00605/) (2019)
- [Making Heads or Tails: Towards Semantically Consistent Visual Counterfactuals](https://mlsanity.com/paper/2203.12892/) (2022)
- [Unlocking Feature Visualization for Deeper Networks with MAgnitude Constrained Optimization](https://mlsanity.com/paper/2306.06805/) (2023)
- [PDiscoNet: Semantically consistent part discovery for fine-grained recognition](https://mlsanity.com/paper/2309.03173/) (2023)
- [PermuteAttack: Counterfactual Explanation of Machine Learning Credit Scorecards](https://mlsanity.com/paper/2008.10138/) (2020)
- [Counterfactual Explanation Based on Gradual Construction for Deep Networks](https://mlsanity.com/paper/2008.01897/) (2020)
- [Interpreting Deep Neural Networks with Relative Sectional Propagation by Analyzing Comparative Gradients and Hostile Activations](https://mlsanity.com/paper/2012.03434/) (2020)
- [Choose your Data Wisely: A Framework for Semantic Counterfactuals](https://mlsanity.com/paper/2305.17667/) (2023)
- [Diffusion-based Visual Counterfactual Explanations -- Towards Systematic Quantitative Evaluation](https://mlsanity.com/paper/2308.06100/) (2023)
- [Generating Post-hoc Explanations for Skip-gram-based Node Embeddings by Identifying Important Nodes with Bridgeness](https://mlsanity.com/paper/2304.12036/) (2023)
- [Towards Better Visualizing the Decision Basis of Networks via Unfold and Conquer Attribution Guidance](https://mlsanity.com/paper/2312.14201/) (2023)

## Abstract (arXiv)

In the domain of non-generative visual counterfactual explanations (CE), traditional techniques frequently involve the substitution of sections within a query image with corresponding sections from distractor images. Such methods have historically overlooked the semantic relevance of the replacement regions to the target object, thereby impairing the model's interpretability and hindering the editing workflow. Addressing these challenges, the present study introduces an innovative methodology named as Weighted Semantic Map with Auto-adaptive Candidate Editing Network (WSAE-Net). Characterized by two significant advancements: the determination of an weighted semantic map and the auto-adaptive candidate editing sequence. First, the generation of the weighted semantic map is designed to maximize the reduction of non-semantic feature units that need to be computed, thereby optimizing computational efficiency. Second, the auto-adaptive candidate editing sequences are designed to determine the optimal computational order among the feature units to be processed, thereby ensuring the efficient generation of counterfactuals while maintaining the semantic relevance of the replacement feature units to the target object. Through comprehensive experimentation, our methodology demonstrates superior performance, contributing to a more lucid and in-depth understanding of visual counterfactual explanations.