# Unfooling Perturbation-Based Post Hoc Explainers

arXiv: https://arxiv.org/abs/2205.14772 · HTML: https://mlsanity.com/paper/2205.14772/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 49 references that are in the map.
- First posted on arXiv 29 May 2022.

## Builds on (most cited first, 18 of 18)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective](https://mlsanity.com/paper/2202.01602/) (2022)
- [Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database](https://mlsanity.com/paper/2011.08512/) (2020)
- [On the Tractability of SHAP Explanations](https://mlsanity.com/paper/2009.08634/) (2020)
- [Explaining the Explainer: A First Theoretical Analysis of LIME](https://mlsanity.com/paper/2001.03447/) (2020)
- [Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contexts](https://mlsanity.com/paper/2201.10295/) (2022)
- [On Locality of Local Explanation Models](https://mlsanity.com/paper/2106.14648/) (2021)
- [Deceptive AI Explanations: Creation and Detection](https://mlsanity.com/paper/2001.07641/) (2020)
- [Backdooring Explainable Machine Learning](https://mlsanity.com/paper/2204.09498/) (2022)
- [A Framework for Evaluating Post Hoc Feature-Additive Explainers](https://mlsanity.com/paper/2106.08376/) (2021)

## Built on by (most cited first, 12 of 12)

- [Adversarial attacks and defenses in explainable artificial intelligence: A survey](https://mlsanity.com/paper/2306.06123/) (2023)
- [SAFARI: Versatile and Efficient Evaluations for Robustness of Interpretability](https://mlsanity.com/paper/2208.09418/) (2022)
- [Pixel-Grounded Prototypical Part Networks](https://mlsanity.com/paper/2309.14531/) (2023)
- [Teaching AI to Teach: Leveraging Limited Human Salience Data Into Unlimited Saliency-Based Training](https://mlsanity.com/paper/2306.05527/) (2023)
- [X Hacking: The Threat of Misguided AutoML](https://mlsanity.com/paper/2401.08513/) (2024)
- [Pixel-level Certified Explanations via Randomized Smoothing](https://mlsanity.com/paper/2506.15499/) (2025)
- [Don't Lie to Me: Avoiding Malicious Explanations with STEALTH](https://mlsanity.com/paper/2301.10407/) (2023)
- [How Well Do Feature-Additive Explainers Explain Feature-Additive Predictors?](https://mlsanity.com/paper/2310.18496/) (2023)
- [Factor Graph-based Interpretable Neural Networks](https://mlsanity.com/paper/2502.14572/) (2025)
- [Explainable but Vulnerable: Adversarial Attacks on XAI Explanation in Cybersecurity Applications](https://mlsanity.com/paper/2510.03623/) (2025)
- [Crushing the Evidence: A Dual-Penalty Evasion Framework for Fooling White-Box Explainable AI Auditors](https://mlsanity.com/paper/2608.00566/) (2026)
- [ExplainGuard: A Zero Trust Framework for Post-Hoc Explanation Integrity Guarantees in Blackbox XAI Models](https://mlsanity.com/paper/2608.21803/) (2026)

## Abstract (arXiv)

Monumental advancements in artificial intelligence (AI) have lured the interest of doctors, lenders, judges, and other professionals. While these high-stakes decision-makers are optimistic about the technology, those familiar with AI systems are wary about the lack of transparency of its decision-making processes. Perturbation-based post hoc explainers offer a model agnostic means of interpreting these systems while only requiring query-level access. However, recent work demonstrates that these explainers can be fooled adversarially. This discovery has adverse implications for auditors, regulators, and other sentinels. With this in mind, several natural questions arise - how can we audit these black box systems? And how can we ascertain that the auditee is complying with the audit in good faith? In this work, we rigorously formalize this problem and devise a defense against adversarial attacks on perturbation-based explainers. We propose algorithms for the detection (CAD-Detect) and defense (CAD-Defend) of these attacks, which are aided by our novel conditional anomaly detection approach, KNN-CAD. We demonstrate that our approach successfully detects whether a black box system adversarially conceals its decision-making process and mitigates the adversarial attack on real-world data for the prevalent explainers, LIME and SHAP.