# Focus! Rating XAI Methods and Finding Biases

arXiv: https://arxiv.org/abs/2109.15035 · HTML: https://mlsanity.com/paper/2109.15035/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 28 Sep 2021.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Top-down Neural Attention by Excitation Backprop](https://mlsanity.com/paper/1608.00507/) (2016)
- [Explaining NonLinear Classification Decisions with Deep Taylor Decomposition](https://mlsanity.com/paper/1512.02479/) (2015)
- [Towards Explainable Artificial Intelligence](https://mlsanity.com/paper/1909.12072/) (2019)
- [Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networks](https://mlsanity.com/paper/1904.00605/) (2019)
- [Investigating Saturation Effects in Integrated Gradients](https://mlsanity.com/paper/2010.12697/) (2020)
- [The MAMe Dataset: On the relevance of High Resolution and Variable Shape image properties](https://mlsanity.com/paper/2007.13693/) (2020)

## Built on by (most cited first, 20 of 20)

- [Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond](https://mlsanity.com/paper/2202.06861/) (2022)
- [The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantus](https://mlsanity.com/paper/2302.07265/) (2023)
- [Finding the right XAI method -- A Guide for the Evaluation and Ranking of Explainable AI Methods in Climate Science](https://mlsanity.com/paper/2303.00652/) (2023)
- [Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanations](https://mlsanity.com/paper/2211.12486/) (2022)
- [Establishing and Evaluating Trustworthy AI: Overview and Research Challenges](https://mlsanity.com/paper/2411.09973/) (2024)
- [Classification Metrics for Image Explanations: Towards Building Reliable XAI-Evaluations](https://mlsanity.com/paper/2406.05068/) (2024)
- [FLEXtime: Filterbank learning to explain time series](https://mlsanity.com/paper/2411.05841/) (2024)
- [On the Evaluation Consistency of Attribution-based Explanations](https://mlsanity.com/paper/2407.19471/) (2024)
- [Red-Teaming Segment Anything Model](https://mlsanity.com/paper/2404.02067/) (2024)
- [Evolutionary approaches to explainable machine learning](https://mlsanity.com/paper/2306.14786/) (2023)
- [Evolutionary Computation and Explainable AI: A Roadmap to Understandable Intelligent Systems](https://mlsanity.com/paper/2406.07811/) (2024)
- [Understanding User Preferences in Explainable Artificial Intelligence: A Survey and a Mapping Function Proposal](https://mlsanity.com/paper/2302.03180/) (2023)
- [Advancing Attribution-Based Neural Network Explainability through Relative Absolute Magnitude Layer-Wise Relevance Propagation and Multi-Component Evaluation](https://mlsanity.com/paper/2412.09311/) (2024)
- [Measuring the Effect of Background on Classification and Feature Importance in Deep Learning for AV Perception](https://mlsanity.com/paper/2512.05937/) (2025)
- [Exploring SAIG Methods for an Objective Evaluation of XAI](https://mlsanity.com/paper/2602.08715/) (2026)
- [Impact of Feature Encoding on Malware Classification Explainability](https://mlsanity.com/paper/2307.05614/) (2023)
- [Towards a Novel Measure of User Trust in XAI Systems](https://mlsanity.com/paper/2405.05766/) (2024)
- [Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understanding](https://mlsanity.com/paper/2603.25251/) (2026)
- [FAIR_XAI: Improving Multimodal Foundation Model Fairness via Explainability for Wellbeing Assessment](https://mlsanity.com/paper/2604.23786/) (2026)
- [Learnable Time-Frequency Masks for Explaining Time-Series Classifiers](https://mlsanity.com/paper/2609.29270/) (2026)

## Abstract (arXiv)

AI explainability improves the transparency of models, making them more trustworthy. Such goals are motivated by the emergence of deep learning models, which are obscure by nature; even in the domain of images, where deep learning has succeeded the most, explainability is still poorly assessed. In the field of image recognition many feature attribution methods have been proposed with the purpose of explaining a model's behavior using visual cues. However, no metrics have been established so far to assess and select these methods objectively. In this paper we propose a consistent evaluation score for feature attribution methods -- the Focus -- designed to quantify their coherency to the task. While most previous work adds out-of-distribution noise to samples, we introduce a methodology to add noise from within the distribution. This is done through mosaics of instances from different classes, and the explanations these generate. On those, we compute a visual pseudo-precision metric, Focus. First, we show the robustness of the approach through a set of randomization experiments. Then we use Focus to compare six popular explainability techniques across several CNN architectures and classification datasets. Our results find some methods to be consistently reliable (LRP, GradCAM), while others produce class-agnostic explanations (SmoothGrad, IG). Finally we introduce another application of Focus, using it for the identification and characterization of biases found in models. This empowers bias-management tools, in another small step towards trustworthy AI.