# What You See is What You Classify: Black Box Attributions

arXiv: https://arxiv.org/abs/2205.11266 · HTML: https://mlsanity.com/paper/2205.11266/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 23 May 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [Real Time Image Saliency for Black Box Classifiers](https://mlsanity.com/paper/1705.07857/) (2017)
- [Explaining Image Classifiers by Counterfactual Generation](https://mlsanity.com/paper/1807.08024/) (2018)
- [What do we need to build explainable AI systems for the medical domain?](https://mlsanity.com/paper/1712.09923/) (2017)
- [Explainable Deep Learning: A Field Guide for the Uninitiated](https://mlsanity.com/paper/2004.14545/) (2020)
- [Black-box Explanation of Object Detectors via Saliency Maps](https://mlsanity.com/paper/2006.03204/) (2020)
- [iGOS++: Integrated Gradient Optimized Saliency by Bilateral Perturbations](https://mlsanity.com/paper/2012.15783/) (2020)

## Built on by (most cited first, 12 of 12)

- [Studying How to Efficiently and Effectively Guide Models with Explanations](https://mlsanity.com/paper/2303.11932/) (2023)
- [Q-SENN: Quantized Self-Explaining Neural Networks](https://mlsanity.com/paper/2312.13839/) (2023)
- [Comprehensive Attribution: Inherently Explainable Vision Model with Feature Detector](https://mlsanity.com/paper/2407.19308/) (2024)
- [A Survey of Explainable AI in Deep Visual Modeling: Methods and Metrics](https://mlsanity.com/paper/2301.13445/) (2023)
- [From Classification to Segmentation with Explainable AI: A Study on Crack Detection and Growth Monitoring](https://mlsanity.com/paper/2309.11267/) (2023)
- [Challenges and Opportunities in Text Generation Explainability](https://mlsanity.com/paper/2405.08468/) (2024)
- [Explainable Image Recognition via Enhanced Slot-attention Based Classifier](https://mlsanity.com/paper/2407.05616/) (2024)
- [Counterfactual Image Generation for adversarially robust and interpretable Classifiers](https://mlsanity.com/paper/2310.00761/) (2023)
- [Why the Agent Made that Decision: Contrastive Explanation Learning for Reinforcement Learning](https://mlsanity.com/paper/2411.16120/) (2024)
- [When, How Long and How Much? Interpretable Neural Networks for Time Series Regression by Learning to Mask and Aggregate](https://mlsanity.com/paper/2512.03578/) (2025)
- [What You Read is What You Classify: Highlighting Attributions to Text and Text-Like Inputs](https://mlsanity.com/paper/2602.24149/) (2026)
- [Building Trust in Artificial Intelligence: A Necessity for Railway Applications](https://mlsanity.com/paper/2609.18278/) (2026)

## Abstract (arXiv)

An important step towards explaining deep image classifiers lies in the identification of image regions that contribute to individual class scores in the model's output. However, doing this accurately is a difficult task due to the black-box nature of such networks. Most existing approaches find such attributions either using activations and gradients or by repeatedly perturbing the input. We instead address this challenge by training a second deep network, the Explainer, to predict attributions for a pre-trained black-box classifier, the Explanandum. These attributions are provided in the form of masks that only show the classifier-relevant parts of an image, masking out the rest. Our approach produces sharper and more boundary-precise masks when compared to the saliency maps generated by other methods. Moreover, unlike most existing approaches, ours is capable of directly generating very distinct class-specific masks in a single forward pass. This makes the proposed method very efficient during inference. We show that our attributions are superior to established methods both visually and quantitatively with respect to the PASCAL VOC-2007 and Microsoft COCO-2014 datasets.