mlsanityData to 14 Oct 2026

SoK: Explainable Machine Learning for Computer Security Applications

arXiv 2208.10605 ↗First posted 22 Aug 2022cs.CR cs.CY cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

Explainable Artificial Intelligence (XAI) aims to improve the transparency of machine learning (ML) pipelines. We systematize the increasingly growing (but fragmented) microcosm of studies that develop and utilize XAI methods for defensive and offensive cybersecurity tasks. We identify 3 cybersecurity stakeholders, i.e., model users, designers, and adversaries, who utilize XAI for 4 distinct objectives within an ML pipeline, namely 1) XAI-enabled user assistance, 2) XAI-enabled model verification, 3) explanation verification & robustness, and 4) offensive use of explanations. Our analysis of the literature indicates that many of the XAI applications are designed with little understanding of how they might be integrated into analyst workflows -- user studies for explanation evaluation are conducted in only 14% of the cases. The security literature sometimes also fails to disentangle the role of the various stakeholders, e.g., by providing explanations to model users and designers while also exposing them to adversaries. Additionally, the role of model designers is particularly minimized in the security literature. To this end, we present an illustrative tutorial for model designers, demonstrating how XAI can help with model verification. We also discuss scenarios where interpretability by design may be a better alternative. The systematization and the tutorial enable us to challenge several assumptions, and present open problems that can help shape the future of XAI research within cybersecurity.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 52 of its 161 references that are in the map. First posted on arXiv 22 Aug 2022.

Builds on

52 of 161 references.

References · 52

  1. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  2. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  3. 2018Sanity Checks for Saliency Mapscited by 1,156
  4. 2016The Mythos of Model Interpretabilitycited by 721
  5. 2017Evasion Attacks against Machine Learning at Test Timecited by 689
  6. 2017Safe Reinforcement Learning via Shieldingcited by 449
  7. 2017Interpretation of Neural Networks is Fragilecited by 444
  8. 2017Wild Patterns: Ten Years After the Rise of Adversarial Machine Learningcited by 414
  9. 2016European Union regulations on algorithmic decision-making and a "right to explanation"cited by 344
  10. 2018On the Robustness of Interpretability Methodscited by 314
  11. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  12. 2019Explanations can be manipulated and geometry is to blamecited by 216
38 more
  1. 2019Explainable Machine Learning for Scientific Insights and Discoveriescited by 166
  2. 2022The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspectivecited by 159
  3. 2020Deep Learning based Vulnerability Detection: Are We There Yet?cited by 144
  4. 2018Automated Vulnerability Detection in Source Code Using Deep Representation Learningcited by 114
  5. 2019Fairwashing: the risk of rationalizationcited by 81
  6. 2018Interpretable Deep Learning under Firecited by 74
  7. 2017Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamplescited by 66
  8. 2021Vulnerability Detection with Fine-grained Interpretationscited by 56
  9. 2020Towards falsifiable interpretability researchcited by 53
  10. 2021Exploiting Explanations for Model Inversion Attackscited by 49
  11. 2021Explainability-based Backdoor Attacks Against Graph Neural Networkscited by 42
  12. 2019Robust Decision Trees Against Adversarial Examplescited by 40
  13. 2019Evaluating Explanation Methods for Deep Learning in Securitycited by 38
  14. 2018Learning Finite State Representations of Recurrent Policy Networkscited by 37
  15. 2020What Do You See? Evaluation of Explainable Artificial Intelligence (XAI) Interpretability through Neural Backdoorscited by 32
  16. 2019Explaining Vulnerabilities of Deep Learning to Adversarial Malware Binariescited by 31
  17. 2018Verification for Machine Learning, Autonomy, and Neural Networks Surveycited by 24
  18. 2020Efficient Training of Robust Decision Trees Against Adversarial Examplescited by 24
  19. 2018An Adversarial Approach for Explainable AI in Intrusion Detection Systemscited by 22
  20. 2022The Role of Machine Learning in Cybersecuritycited by 22
  21. 2018Recurrent Neural Network Attention Mechanisms for Interpretable System Log Anomaly Detectioncited by 21
  22. 2022Attribution-based Explanations that Provide Recourse Cannot be Robustcited by 20
  23. 2021A Survey on Data-driven Software Vulnerability Assessment and Prioritizationcited by 18
  24. 2021DeepAID: Interpreting and Improving Deep Learning-based Anomaly Detection in Security Applicationscited by 18
  25. 2020Why an Android App is Classified as Malware? Towards Malware Classification Interpretationcited by 16
  26. 2018Explainable Securitycited by 15
  27. 2019Towards Quantification of Explainability in Explainable Artificial Intelligence Methodscited by 15
  28. 2017DeltaPhish: Detecting Phishing Webpages in Compromised Websitescited by 13
  29. 2019Treant: Training Evasion-Aware Decision Treescited by 12
  30. 2019The Price of Interpretabilitycited by 10
  31. 2017Adversarial-Playground: A Visualization Suite Showing How Adversarial Examples Fool Deep Learningcited by 8
  32. 2019Proposed Guidelines for the Responsible Use of Explainable Machine Learningcited by 8
  33. 2021On the Importance of Domain-specific Explanations in AI-based Cybersecurity Systems (Technical Report)cited by 7
  34. 2022Learning to be adversarially robust and differentially privatecited by 6
  35. 2019VizADS-B: Analyzing Sequences of ADS-B Images Using Explainable Convolutional LSTM Encoder-Decoder to Detect Cyber Attackscited by 5
  36. 2019Towards Self-Explainable Cyber-Physical Systemscited by 5
  37. 2021To Trust or Not to Trust a Regressor: Estimating and Explaining Trustworthiness of Regression Predictionscited by 5
  38. 2021Research trends, challenges, and emerging topics of digital forensics: A review of reviewscited by 5

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 12; 2025: 4; 2026: 1.

20 papers.

Sort:

  1. 2024Can LLMs Understand Computer Networks? Towards a Virtual System Administratorcited by 11
  2. 2024A Survey of Malware Detection Using Deep Learningcited by 9
  3. 2023MalProtect: Stateful Defense Against Adversarial Query Attacks in ML-based Malware Detectioncited by 8
  4. 2024Exploring the Efficacy of Large Language Models (GPT-4) in Binary Reverse Engineeringcited by 8
  5. 2022SoK: Modeling Explainability in Security Analytics for Interpretability, Trustworthiness, and Usabilitycited by 7
  6. 2023Rethinking the Trigger-injecting Position in Graph Backdoor Attackcited by 4
  7. 2024More Questions than Answers? Lessons from Integrating Explainable AI into a Cyber-AI Toolcited by 3
  8. 2024XG-NID: Dual-Modality Network Intrusion Detection using a Heterogeneous Graph Neural Network and Large Language Modelcited by 3
  9. 2024A Systematic Literature Review on Explainability for Machine/Deep Learning-based Software Engineering Researchcited by 2
  10. 2024A Sim2Real Approach for Identifying Task-Relevant Properties in Interpretable Machine Learningcited by 2
  11. 2024SoK: Security and Privacy Risks of Healthcare AIcited by 2
  12. 2024Contextualized AI for Cyber Defense: An Automated Survey using LLMscited by 1
8 more
  1. 2024Comparative Analysis of Black-Box and White-Box Machine Learning Model in Phishing Detectioncited by 1
  2. 2025Too Much to Trust? Measuring the Security and Cognitive Impacts of Explainability in AI-Driven SOCscited by 1
  3. 2024A Modular Approach to Automatic Cyber Threat Attribution using Opinion Poolscited by 0
  4. 2024State Frequency Estimation for Anomaly Detectioncited by 0
  5. 2025Policy alone is probably not the solution: A large-scale experiment on how developers struggle to design meaningful end-user explanationscited by 0
  6. 2025On the Effect of Ruleset Tuning and Data Imbalance on Explainable Network Security Alert Classifications: a Case-Study on DeepCASEcited by 0
  7. 2025Towards a General Framework for Predicting and Explaining the Hardness of Graph-based Combinatorial Optimization Problems using Machine Learning and Association Rule Miningcited by 0
  8. 2026SoK: Privacy Attacks on Machine Learning via Explainable AIcited by 0