# What Do You See? Evaluation of Explainable Artificial Intelligence (XAI) Interpretability through Neural Backdoors

arXiv: https://arxiv.org/abs/2009.10639 · HTML: https://mlsanity.com/paper/2009.10639/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2020.

## Built on by (most cited first, 30 of 32)

- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond](https://mlsanity.com/paper/2103.10689/) (2021)
- [Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis](https://mlsanity.com/paper/2211.02408/) (2022)
- [SoK: Explainable Machine Learning for Computer Security Applications](https://mlsanity.com/paper/2208.10605/) (2022)
- [Backdoor Attack with Sparse and Invisible Trigger](https://mlsanity.com/paper/2306.06209/) (2023)
- [A Survey of Neural Trojan Attacks and Defenses in Deep Learning](https://mlsanity.com/paper/2202.07183/) (2022)
- [Explainable Artificial Intelligence for Medical Applications: A Review](https://mlsanity.com/paper/2412.01829/) (2024)
- [Towards A Holistic View of Bias in Machine Learning: Bridging Algorithmic Fairness and Imbalanced Learning](https://mlsanity.com/paper/2207.06084/) (2022)
- [Can We Trust Explainable AI Methods on ASR? An Evaluation on Phoneme Recognition](https://mlsanity.com/paper/2305.18011/) (2023)
- [A Survey of Malware Detection Using Deep Learning](https://mlsanity.com/paper/2407.19153/) (2024)
- [Explainable Artificial Intelligence (XAI): An Engineering Perspective](https://mlsanity.com/paper/2101.03613/) (2021)
- [Cross-Model Consensus of Explanations and Beyond for Image Classification Models: An Empirical Study](https://mlsanity.com/paper/2109.00707/) (2021)
- [Ethical Framework for Harnessing the Power of AI in Healthcare and Beyond](https://mlsanity.com/paper/2309.00064/) (2023)
- [Explainable Artificial Intelligence Methods in Combating Pandemics: A Systematic Review](https://mlsanity.com/paper/2112.12705/) (2021)
- [Auditing Visualizations: Transparency Methods Struggle to Detect Anomalous Behavior](https://mlsanity.com/paper/2206.13498/) (2022)
- [Detection Accuracy for Evaluating Compositional Explanations of Units](https://mlsanity.com/paper/2109.07804/) (2021)
- [IBO: Inpainting-Based Occlusion to Enhance Explainable Artificial Intelligence Evaluation in Histopathology](https://mlsanity.com/paper/2408.16395/) (2024)
- [TRAPDOOR: Repurposing backdoors to detect dataset bias in machine learning-based genomic analysis](https://mlsanity.com/paper/2108.10132/) (2021)
- [Forward Learning for Gradient-based Black-box Saliency Map Generation](https://mlsanity.com/paper/2403.15603/) (2024)
- [A Backdoor-based Explainable AI Benchmark for High Fidelity Evaluation of Attributions](https://mlsanity.com/paper/2405.02344/) (2024)
- [Position: Explainability Research Must Prioritize Foundations over Ad-hoc Methods](https://mlsanity.com/paper/2607.14123/) (2026)
- [Understanding User Preferences in Explainable Artificial Intelligence: A Survey and a Mapping Function Proposal](https://mlsanity.com/paper/2302.03180/) (2023)
- [Inpainting the Gaps: A Novel Framework for Evaluating Explanation Methods in Vision Transformers](https://mlsanity.com/paper/2406.11534/) (2024)
- [Applications of Explainable artificial intelligence in Earth system science](https://mlsanity.com/paper/2406.11882/) (2024)
- [Revocable Backdoor for Deep Model Trading](https://mlsanity.com/paper/2408.00255/) (2024)
- [XAI-Units: Benchmarking Explainability Methods with Unit Tests](https://mlsanity.com/paper/2506.01059/) (2025)
- [Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Models](https://mlsanity.com/paper/2605.19227/) (2026)
- [What's meant by explainable model: A Scoping Review](https://mlsanity.com/paper/2307.09673/) (2023)
- [Manipulating Trajectory Prediction with Backdoors](https://mlsanity.com/paper/2312.13863/) (2023)
- [Knowledge-Augmented Explainable and Interpretable Learning for Anomaly Detection and Diagnosis](https://mlsanity.com/paper/2412.00146/) (2024)

## Abstract (arXiv)

EXplainable AI (XAI) methods have been proposed to interpret how a deep neural network predicts inputs through model saliency explanations that highlight the parts of the inputs deemed important to arrive a decision at a specific target. However, it remains challenging to quantify correctness of their interpretability as current evaluation approaches either require subjective input from humans or incur high computation cost with automated evaluation. In this paper, we propose backdoor trigger patterns--hidden malicious functionalities that cause misclassification--to automate the evaluation of saliency explanations. Our key observation is that triggers provide ground truth for inputs to evaluate whether the regions identified by an XAI method are truly relevant to its output. Since backdoor triggers are the most important features that cause deliberate misclassification, a robust XAI method should reveal their presence at inference time. We introduce three complementary metrics for systematic evaluation of explanations that an XAI method generates and evaluate seven state-of-the-art model-free and model-specific posthoc methods through 36 models trojaned with specifically crafted triggers using color, shape, texture, location, and size. We discovered six methods that use local explanation and feature relevance fail to completely highlight trigger regions, and only a model-free approach can uncover the entire trigger region.