# SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detection

arXiv: https://arxiv.org/abs/2305.00550 · HTML: https://mlsanity.com/paper/2305.00550/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 215 references that are in the map.
- First posted on arXiv 30 Apr 2023.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [One pixel attack for fooling deep neural networks](https://mlsanity.com/paper/1710.08864/) (2017)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Learning under Concept Drift: A Review](https://mlsanity.com/paper/2004.05785/) (2020)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [On Evaluating Adversarial Robustness](https://mlsanity.com/paper/1902.06705/) (2019)
- [Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning](https://mlsanity.com/paper/1712.03141/) (2017)
- [Explainable Machine Learning in Deployment](https://mlsanity.com/paper/1909.06342/) (2019)
- [To Trust Or Not To Trust A Classifier](https://mlsanity.com/paper/1805.11783/) (2018)
- [Challenges in Deploying Machine Learning: a Survey of Case Studies](https://mlsanity.com/paper/2011.09926/) (2020)
- [Kitsune: An Ensemble of Autoencoders for Online Network Intrusion Detection](https://mlsanity.com/paper/1802.09089/) (2018)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time](https://mlsanity.com/paper/1807.07838/) (2018)
- ["Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice](https://mlsanity.com/paper/2212.14315/) (2022)
- [Best Practices for Scientific Research on Neural Architecture Search](https://mlsanity.com/paper/1909.02453/) (2019)
- [UNICORN: Runtime Provenance-Based Detector for Advanced Persistent Threats](https://mlsanity.com/paper/2001.01525/) (2020)
- [Yes, Machine Learning Can Be More Secure! A Case Study on Android Malware Detection](https://mlsanity.com/paper/1704.08996/) (2017)
- [Requirements Engineering for Machine Learning: Perspectives from Data Scientists](https://mlsanity.com/paper/1908.04674/) (2019)
- [A Survey of Network-based Intrusion Detection Data Sets](https://mlsanity.com/paper/1903.02460/) (2019)
- [NetFlow Datasets for Machine Learning-based Network Intrusion Detection Systems](https://mlsanity.com/paper/2011.09144/) (2020)
- [HOLMES: Real-time APT Detection through Correlation of Suspicious Information Flows](https://mlsanity.com/paper/1810.01594/) (2018)
- [Modeling Realistic Adversarial Attacks against Network Intrusion Detection Systems](https://mlsanity.com/paper/2106.09380/) (2021)
- [Survey of Machine Learning Techniques for Malware Analysis](https://mlsanity.com/paper/1710.08189/) (2017)
- [The Role of Machine Learning in Cybersecurity](https://mlsanity.com/paper/2206.09707/) (2022)
- [Realtime Robust Malicious Traffic Detection via Frequency Domain Analysis](https://mlsanity.com/paper/2106.14707/) (2021)
- [Reviewer Integration and Performance Measurement for Malware Detection](https://mlsanity.com/paper/1510.07338/) (2015)
- [Improving Robustness of ML Classifiers against Realizable Evasion Attacks Using Conserved Features](https://mlsanity.com/paper/1708.08327/) (2017)
- [DeepAID: Interpreting and Improving Deep Learning-based Anomaly Detection in Security Applications](https://mlsanity.com/paper/2109.11495/) (2021)
- [A Framework for Cluster and Classifier Evaluation in the Absence of Reference Labels](https://mlsanity.com/paper/2109.11126/) (2021)
- [SoK: The Impact of Unlabelled Data in Cyberthreat Detection](https://mlsanity.com/paper/2205.08944/) (2022)

## Built on by (most cited first, 13 of 13)

- [SoK: On the Offensive Potential of AI](https://mlsanity.com/paper/2412.18442/) (2024)
- [On the use of neurosymbolic AI for defending against cyber attacks](https://mlsanity.com/paper/2408.04996/) (2024)
- [Not All Explanations for Deep Learning Phenomena Are Equally Valuable](https://mlsanity.com/paper/2506.23286/) (2025)
- [Expectations Versus Reality: Evaluating Intrusion Detection Systems in Practice](https://mlsanity.com/paper/2403.17458/) (2024)
- [SoK: Security and Privacy Risks of Healthcare AI](https://mlsanity.com/paper/2409.07415/) (2024)
- [Temporal Analysis of NetFlow Datasets for Network Intrusion Detection Systems](https://mlsanity.com/paper/2503.04404/) (2025)
- [PHASE: Passive Human Activity Simulation Evaluation](https://mlsanity.com/paper/2507.13505/) (2025)
- [Assessing the Impact of a Supervised Classification Filter on Flow-based Hybrid Network Anomaly Detection](https://mlsanity.com/paper/2310.06656/) (2023)
- [Improving Generalization on Cybersecurity Tasks with Multi-Modal Contrastive Learning](https://mlsanity.com/paper/2603.20181/) (2026)
- [Stream Assembly Is an Uncontrolled Treatment in Streaming Intrusion-Detection Benchmarks](https://mlsanity.com/paper/2605.24696/) (2026)
- [IstGPT: LLM-based Anomaly Detection for Spatial-Temporal Graph in Industrial Systems](https://mlsanity.com/paper/2606.01691/) (2026)
- [How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection](https://mlsanity.com/paper/2608.01454/) (2026)
- [Candidate Comparability Before Promotion: Conditional Validation in Adaptive Network Intrusion Detection](https://mlsanity.com/paper/2609.04388/) (2026)

## Abstract (arXiv)

Machine Learning (ML) has become a valuable asset to solve many real-world tasks. For Network Intrusion Detection (NID), however, scientific advances in ML are still seen with skepticism by practitioners. This disconnection is due to the intrinsically limited scope of research papers, many of which primarily aim to demonstrate new methods ``outperforming'' prior work -- oftentimes overlooking the practical implications for deploying the proposed solutions in real systems. Unfortunately, the value of ML for NID depends on a plethora of factors, such as hardware, that are often neglected in scientific literature. This paper aims to reduce the practitioners' skepticism towards ML for NID by "changing" the evaluation methodology adopted in research. After elucidating which "factors" influence the operational deployment of ML in NID, we propose the notion of "pragmatic assessment", which enable practitioners to gauge the real value of ML methods for NID. Then, we show that the state-of-research hardly allows one to estimate the value of ML for NID. As a constructive step forward, we carry out a pragmatic assessment. We re-assess existing ML methods for NID, focusing on the classification of malicious network traffic, and consider: hundreds of configuration settings; diverse adversarial scenarios; and four hardware platforms. Our large and reproducible evaluations enable estimating the quality of ML for NID. We also validate our claims through a user-study with security practitioners.