# SoK: Explainable Machine Learning for Computer Security Applications

arXiv: https://arxiv.org/abs/2208.10605 · HTML: https://mlsanity.com/paper/2208.10605/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 161 references that are in the map.
- First posted on arXiv 22 Aug 2022.

## Builds on (most cited first, 30 of 52)

- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Evasion Attacks against Machine Learning at Test Time](https://mlsanity.com/paper/1708.06131/) (2017)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning](https://mlsanity.com/paper/1712.03141/) (2017)
- [European Union regulations on algorithmic decision-making and a "right to explanation"](https://mlsanity.com/paper/1606.08813/) (2016)
- [On the Robustness of Interpretability Methods](https://mlsanity.com/paper/1806.08049/) (2018)
- [Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients](https://mlsanity.com/paper/1711.09404/) (2017)
- [Explanations can be manipulated and geometry is to blame](https://mlsanity.com/paper/1906.07983/) (2019)
- [Explainable Machine Learning for Scientific Insights and Discoveries](https://mlsanity.com/paper/1905.08883/) (2019)
- [The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective](https://mlsanity.com/paper/2202.01602/) (2022)
- [Deep Learning based Vulnerability Detection: Are We There Yet?](https://mlsanity.com/paper/2009.07235/) (2020)
- [Automated Vulnerability Detection in Source Code Using Deep Representation Learning](https://mlsanity.com/paper/1807.04320/) (2018)
- [Fairwashing: the risk of rationalization](https://mlsanity.com/paper/1901.09749/) (2019)
- [Interpretable Deep Learning under Fire](https://mlsanity.com/paper/1812.00891/) (2018)
- [Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples](https://mlsanity.com/paper/1711.09576/) (2017)
- [Vulnerability Detection with Fine-grained Interpretations](https://mlsanity.com/paper/2106.10478/) (2021)
- [Towards falsifiable interpretability research](https://mlsanity.com/paper/2010.12016/) (2020)
- [Exploiting Explanations for Model Inversion Attacks](https://mlsanity.com/paper/2104.12669/) (2021)
- [Explainability-based Backdoor Attacks Against Graph Neural Networks](https://mlsanity.com/paper/2104.03674/) (2021)
- [Robust Decision Trees Against Adversarial Examples](https://mlsanity.com/paper/1902.10660/) (2019)
- [Evaluating Explanation Methods for Deep Learning in Security](https://mlsanity.com/paper/1906.02108/) (2019)
- [Learning Finite State Representations of Recurrent Policy Networks](https://mlsanity.com/paper/1811.12530/) (2018)
- [What Do You See? Evaluation of Explainable Artificial Intelligence (XAI) Interpretability through Neural Backdoors](https://mlsanity.com/paper/2009.10639/) (2020)
- [Explaining Vulnerabilities of Deep Learning to Adversarial Malware Binaries](https://mlsanity.com/paper/1901.03583/) (2019)
- [Verification for Machine Learning, Autonomy, and Neural Networks Survey](https://mlsanity.com/paper/1810.01989/) (2018)
- [Efficient Training of Robust Decision Trees Against Adversarial Examples](https://mlsanity.com/paper/2012.10438/) (2020)

## Built on by (most cited first, 20 of 20)

- [Can LLMs Understand Computer Networks? Towards a Virtual System Administrator](https://mlsanity.com/paper/2404.12689/) (2024)
- [A Survey of Malware Detection Using Deep Learning](https://mlsanity.com/paper/2407.19153/) (2024)
- [MalProtect: Stateful Defense Against Adversarial Query Attacks in ML-based Malware Detection](https://mlsanity.com/paper/2302.10739/) (2023)
- [Exploring the Efficacy of Large Language Models (GPT-4) in Binary Reverse Engineering](https://mlsanity.com/paper/2406.06637/) (2024)
- [SoK: Modeling Explainability in Security Analytics for Interpretability, Trustworthiness, and Usability](https://mlsanity.com/paper/2210.17376/) (2022)
- [Rethinking the Trigger-injecting Position in Graph Backdoor Attack](https://mlsanity.com/paper/2304.02277/) (2023)
- [More Questions than Answers? Lessons from Integrating Explainable AI into a Cyber-AI Tool](https://mlsanity.com/paper/2408.04746/) (2024)
- [XG-NID: Dual-Modality Network Intrusion Detection using a Heterogeneous Graph Neural Network and Large Language Model](https://mlsanity.com/paper/2408.16021/) (2024)
- [A Systematic Literature Review on Explainability for Machine/Deep Learning-based Software Engineering Research](https://mlsanity.com/paper/2401.14617/) (2024)
- [A Sim2Real Approach for Identifying Task-Relevant Properties in Interpretable Machine Learning](https://mlsanity.com/paper/2406.00116/) (2024)
- [SoK: Security and Privacy Risks of Healthcare AI](https://mlsanity.com/paper/2409.07415/) (2024)
- [Contextualized AI for Cyber Defense: An Automated Survey using LLMs](https://mlsanity.com/paper/2409.13524/) (2024)
- [Comparative Analysis of Black-Box and White-Box Machine Learning Model in Phishing Detection](https://mlsanity.com/paper/2412.02084/) (2024)
- [Too Much to Trust? Measuring the Security and Cognitive Impacts of Explainability in AI-Driven SOCs](https://mlsanity.com/paper/2503.02065/) (2025)
- [A Modular Approach to Automatic Cyber Threat Attribution using Opinion Pools](https://mlsanity.com/paper/2401.14090/) (2024)
- [State Frequency Estimation for Anomaly Detection](https://mlsanity.com/paper/2412.03442/) (2024)
- [Policy alone is probably not the solution: A large-scale experiment on how developers struggle to design meaningful end-user explanations](https://mlsanity.com/paper/2503.15512/) (2025)
- [On the Effect of Ruleset Tuning and Data Imbalance on Explainable Network Security Alert Classifications: a Case-Study on DeepCASE](https://mlsanity.com/paper/2507.01571/) (2025)
- [Towards a General Framework for Predicting and Explaining the Hardness of Graph-based Combinatorial Optimization Problems using Machine Learning and Association Rule Mining](https://mlsanity.com/paper/2512.20915/) (2025)
- [SoK: Privacy Attacks on Machine Learning via Explainable AI](https://mlsanity.com/paper/2609.10627/) (2026)

## Abstract (arXiv)

Explainable Artificial Intelligence (XAI) aims to improve the transparency of machine learning (ML) pipelines. We systematize the increasingly growing (but fragmented) microcosm of studies that develop and utilize XAI methods for defensive and offensive cybersecurity tasks. We identify 3 cybersecurity stakeholders, i.e., model users, designers, and adversaries, who utilize XAI for 4 distinct objectives within an ML pipeline, namely 1) XAI-enabled user assistance, 2) XAI-enabled model verification, 3) explanation verification & robustness, and 4) offensive use of explanations. Our analysis of the literature indicates that many of the XAI applications are designed with little understanding of how they might be integrated into analyst workflows -- user studies for explanation evaluation are conducted in only 14% of the cases. The security literature sometimes also fails to disentangle the role of the various stakeholders, e.g., by providing explanations to model users and designers while also exposing them to adversaries. Additionally, the role of model designers is particularly minimized in the security literature. To this end, we present an illustrative tutorial for model designers, demonstrating how XAI can help with model verification. We also discuss scenarios where interpretability by design may be a better alternative. The systematization and the tutorial enable us to challenge several assumptions, and present open problems that can help shape the future of XAI research within cybersecurity.