# PhaseWin: An Efficient Search Algorithm for Faithful Visual Attribution

arXiv: https://arxiv.org/abs/2606.18008 · HTML: https://mlsanity.com/paper/2606.18008/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 45 of its 67 references that are in the map.
- First posted on arXiv 16 Jun 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [Problems with Shapley-value-based explanations as feature importance measures](https://mlsanity.com/paper/2002.11097/) (2020)
- [Interpreting CLIP's Image Representation via Text-Based Decomposition](https://mlsanity.com/paper/2310.05916/) (2023)
- [MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs](https://mlsanity.com/paper/2502.17422/) (2025)
- [Smooth Grad-CAM++: An Enhanced Inference Level Visualization Technique for Deep Convolutional Neural Network Models](https://mlsanity.com/paper/1908.01224/) (2019)
- [A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving](https://mlsanity.com/paper/2011.10671/) (2020)
- [Lazier Than Lazy Greedy](https://mlsanity.com/paper/1409.7938/) (2014)
- [Black-box Explanation of Object Detectors via Saliency Maps](https://mlsanity.com/paper/2006.03204/) (2020)
- [Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis](https://mlsanity.com/paper/2111.04138/) (2021)
- [Explain Any Concept: Segment Anything Meets Concept-Based Explanation](https://mlsanity.com/paper/2305.10289/) (2023)
- [Going Beyond XAI: A Systematic Survey for Explanation-Guided Learning](https://mlsanity.com/paper/2212.03954/) (2022)
- [Gradient Methods for Submodular Maximization](https://mlsanity.com/paper/1708.03949/) (2017)
- [Making Sense of Dependence: Efficient Black-box Explanations Using Dependence Measure](https://mlsanity.com/paper/2206.06219/) (2022)
- [SAFE: Sensitivity-Aware Features for Out-of-Distribution Object Detection](https://mlsanity.com/paper/2208.13930/) (2022)
- [One Explanation is Not Enough: Structured Attention Graphs for Image Classification](https://mlsanity.com/paper/2011.06733/) (2020)
- [Token Activation Map to Visually Explain Multimodal LLMs](https://mlsanity.com/paper/2506.23270/) (2025)

## Built on by (most cited first, 1 of 1)

- [A Good Initialization is All You Need for Faithful Visual Attribution](https://mlsanity.com/paper/2607.06726/) (2026)

## Abstract (arXiv)

Visual attribution is a fundamental tool for interpreting modern vision and vision-language models, particularly when their decisions must be inspected, diagnosed, or audited. Its goal is to explain how a model's decision depends on local regions of the visual input, typically by assigning an importance ordering over candidate image regions. Given an image partitioned into n regions, faithful attribution can be cast as an ordered subset-search problem, in which progressively inserting the selected regions should recover the target model response as early as possible. Exhaustive search over region subsets incurs exponential cost, while the widely used greedy search still requires a quadratic number of model evaluations, because every selection step rescores all remaining candidates. We propose PhaseWin, an efficient subset-search algorithm for faithful visual attribution. PhaseWin reorganizes greedy region selection into a phased window-search procedure: rather than re-evaluating the full candidate set at every step, it alternates between global candidate screening, adaptive pruning, and localized window refinement, while preserving the essential region-ranking behavior of greedy search. We analyze PhaseWin under monotone evidence-accumulation conditions and show that, under feature-level structural assumptions, it attains controllable linear evaluation complexity together with near-greedy faithfulness guarantees. Extensive experiments on image classification, object detection, visual grounding, and image captioning show that, among all compared attribution methods, PhaseWin reaches high faithfulness with the fewest forward passes, empirically realizing the predicted reduction from O(n2) to O(n). The code is available at https://github.com/Qihuai27/phasewin-va.