# DeepAID: Interpreting and Improving Deep Learning-based Anomaly Detection in Security Applications

arXiv: https://arxiv.org/abs/2109.11495 · HTML: https://mlsanity.com/paper/2109.11495/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 68 references that are in the map.
- First posted on arXiv 23 Sep 2021.

## Builds on (most cited first, 30 of 38)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [Explaining NonLinear Classification Decisions with Deep Taylor Decomposition](https://mlsanity.com/paper/1512.02479/) (2015)
- [Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications](https://mlsanity.com/paper/1802.03903/) (2018)
- [Explaining Image Classifiers by Counterfactual Generation](https://mlsanity.com/paper/1807.08024/) (2018)
- [Kitsune: An Ensemble of Autoencoders for Online Network Intrusion Detection](https://mlsanity.com/paper/1802.09089/) (2018)
- [Adversarially Learned Anomaly Detection](https://mlsanity.com/paper/1812.02288/) (2018)
- [Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks](https://mlsanity.com/paper/1602.03616/) (2016)
- [Interpretable Deep Learning under Fire](https://mlsanity.com/paper/1812.00891/) (2018)
- [Beyond Sparsity: Tree Regularization of Deep Models for Interpretability](https://mlsanity.com/paper/1711.06178/) (2017)
- [UNICORN: Runtime Provenance-Based Detector for Advanced Persistent Threats](https://mlsanity.com/paper/2001.01525/) (2020)
- [Evaluating Explanation Methods for Deep Learning in Security](https://mlsanity.com/paper/1906.02108/) (2019)
- [Towards Explanation of DNN-based Prediction with Guided Feature Inversion](https://mlsanity.com/paper/1804.00506/) (2018)

## Built on by (most cited first, 18 of 18)

- [Kairos: Practical Intrusion Detection and Investigation using Whole-system Provenance](https://mlsanity.com/paper/2308.05034/) (2023)
- [Large Language Models can Deliver Accurate and Interpretable Time Series Anomaly Detection](https://mlsanity.com/paper/2405.15370/) (2024)
- [SoK: Explainable Machine Learning for Computer Security Applications](https://mlsanity.com/paper/2208.10605/) (2022)
- [SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detection](https://mlsanity.com/paper/2305.00550/) (2023)
- [FINER: Enhancing State-of-the-art Classifiers with Feature Attribution to Facilitate Security Analysis](https://mlsanity.com/paper/2308.05362/) (2023)
- [SoK: Modeling Explainability in Security Analytics for Interpretability, Trustworthiness, and Usability](https://mlsanity.com/paper/2210.17376/) (2022)
- [AI for DevSecOps: A Landscape and Future Opportunities](https://mlsanity.com/paper/2404.04839/) (2024)
- [XG-NID: Dual-Modality Network Intrusion Detection using a Heterogeneous Graph Neural Network and Large Language Model](https://mlsanity.com/paper/2408.16021/) (2024)
- [AnoGAN for Tabular Data: A Novel Approach to Anomaly Detection](https://mlsanity.com/paper/2405.03075/) (2024)
- [Visually Analyze SHAP Plots to Diagnose Misclassifications in ML-based Intrusion Detection](https://mlsanity.com/paper/2411.02670/) (2024)
- [AnomalyAID: Reliable Interpretation for Semi-supervised Network Anomaly Detection](https://mlsanity.com/paper/2411.11293/) (2024)
- [A Comparative Analysis of DNN-based White-Box Explainable AI Methods in Network Security](https://mlsanity.com/paper/2501.07801/) (2025)
- [UniNet: A Unified Multi-granular Traffic Modeling Framework for Network Security](https://mlsanity.com/paper/2503.04174/) (2025)
- [XAI-based Feature Selection for Improved Network Intrusion Detection Systems](https://mlsanity.com/paper/2410.10050/) (2024)
- [Adaptive NAD: Online and Self-adaptive Unsupervised Network Anomaly Detector](https://mlsanity.com/paper/2410.22967/) (2024)
- [SRLR: Symbolic Regression based Logic Recovery to Counter Programmable Logic Controller Attacks](https://mlsanity.com/paper/2512.11298/) (2025)
- [Spatio-Temporal Attention Graph Neural Network: Explaining Causalities With Attention](https://mlsanity.com/paper/2603.10676/) (2026)
- [RankShift: In-Database Detection and Explanation of Categorical Shifts](https://mlsanity.com/paper/2608.28922/) (2026)

## Abstract (arXiv)

Unsupervised Deep Learning (DL) techniques have been widely used in various security-related anomaly detection applications, owing to the great promise of being able to detect unforeseen threats and superior performance provided by Deep Neural Networks (DNN). However, the lack of interpretability creates key barriers to the adoption of DL models in practice. Unfortunately, existing interpretation approaches are proposed for supervised learning models and/or non-security domains, which are unadaptable for unsupervised DL models and fail to satisfy special requirements in security domains. In this paper, we propose DeepAID, a general framework aiming to (1) interpret DL-based anomaly detection systems in security domains, and (2) improve the practicality of these systems based on the interpretations. We first propose a novel interpretation method for unsupervised DNNs by formulating and solving well-designed optimization problems with special constraints for security domains. Then, we provide several applications based on our Interpreter as well as a model-based extension Distiller to improve security systems by solving domain-specific problems. We apply DeepAID over three types of security-related anomaly detection systems and extensively evaluate our Interpreter with representative prior works. Experimental results show that DeepAID can provide high-quality interpretations for unsupervised DL models while meeting the special requirements of security domains. We also provide several use cases to show that DeepAID can help security operators to understand model decisions, diagnose system mistakes, give feedback to models, and reduce false positives.