# Black-Box Anomaly Attribution

arXiv: https://arxiv.org/abs/2305.18440 · HTML: https://mlsanity.com/paper/2305.18440/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 57 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 20 of 20)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [The many Shapley values for model explanation](https://mlsanity.com/paper/1908.08474/) (2019)
- [A Survey on the Explainability of Supervised Machine Learning](https://mlsanity.com/paper/2011.07876/) (2020)
- [Problems with Shapley-value-based explanations as feature importance measures](https://mlsanity.com/paper/2002.11097/) (2020)
- [Adversarial Robustness through Local Linearization](https://mlsanity.com/paper/1907.02610/) (2019)
- [Do Feature Attribution Methods Correctly Attribute Features?](https://mlsanity.com/paper/2104.14403/) (2021)
- [A Survey on Explainable Anomaly Detection](https://mlsanity.com/paper/2210.06959/) (2022)
- [Interpretable, Multidimensional, Multimodal Anomaly Detection with Negative Sampling for Detection of Device Failure](https://mlsanity.com/paper/2007.10088/) (2020)
- [Attribution-based XAI Methods in Computer Vision: A Review](https://mlsanity.com/paper/2211.14736/) (2022)
- [A Unified Taylor Framework for Revisiting Attribution Methods](https://mlsanity.com/paper/2008.09695/) (2020)
- [A general-purpose method for applying Explainable AI for Anomaly Detection](https://mlsanity.com/paper/2207.11564/) (2022)
- [Anomaly Attribution with Likelihood Compensation](https://mlsanity.com/paper/2208.10679/) (2022)
- [ACE -- An Anomaly Contribution Explainer for Cyber-Security Applications](https://mlsanity.com/paper/1912.00314/) (2019)

## Built on by (most cited first, 1 of 1)

- [Generative Perturbation Analysis for Probabilistic Black-Box Anomaly Attribution](https://mlsanity.com/paper/2308.04708/) (2023)

## Abstract (arXiv)

When the prediction of a black-box machine learning model deviates from the true observation, what can be said about the reason behind that deviation? This is a fundamental and ubiquitous question that the end user in a business or industrial AI application often asks. The deviation may be due to a sub-optimal black-box model, or it may be simply because the sample in question is an outlier. In either case, one would ideally wish to obtain some form of attribution score -- a value indicative of the extent to which an input variable is responsible for the anomaly. In the present paper we address this task of ``anomaly attribution,'' particularly in the setting in which the model is black-box and the training data are not available. Specifically, we propose a novel likelihood-based attribution framework we call the ``likelihood compensation (LC),'' in which the responsibility score is equated with the correction on each input variable needed to attain the highest possible likelihood. We begin by showing formally why mainstream model-agnostic explanation methods, such as the local linear surrogate modeling and Shapley values, are not designed to explain anomalies. In particular, we show that they are ``deviation-agnostic,'' namely, that their explanations are blind to the fact that there is a deviation in the model prediction for the sample of interest. We do this by positioning these existing methods under the unified umbrella of a function family we call the ``integrated gradient family.'' We validate the effectiveness of the proposed LC approach using publicly available data sets. We also conduct a case study with a real-world building energy prediction task and confirm its usefulness in practice based on expert feedback.