# Expectations Versus Reality: Evaluating Intrusion Detection Systems in Practice

arXiv: https://arxiv.org/abs/2403.17458 · HTML: https://mlsanity.com/paper/2403.17458/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 61 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 8 of 8)

- [NetFlow Datasets for Machine Learning-based Network Intrusion Detection Systems](https://mlsanity.com/paper/2011.09144/) (2020)
- [IoT Data Analytics in Dynamic Environments: From An Automated Machine Learning Perspective](https://mlsanity.com/paper/2209.08018/) (2022)
- [SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detection](https://mlsanity.com/paper/2305.00550/) (2023)
- [AOC-IDS: Autonomous Online Framework with Contrastive Learning for Intrusion Detection](https://mlsanity.com/paper/2402.01807/) (2024)
- [Applying Self-supervised Learning to Network Intrusion Detection for Network Flows with Graph Neural Network](https://mlsanity.com/paper/2403.01501/) (2024)
- [I-SiamIDS: an improved Siam-IDS for handling class imbalance in network-based intrusion detection systems](https://mlsanity.com/paper/2009.10940/) (2020)
- [An Intrusion Detection System based on Deep Belief Networks](https://mlsanity.com/paper/2207.02117/) (2022)
- [RIDS : Real-time Intrusion Detection System for WPA3 enabled Enterprise Networks](https://mlsanity.com/paper/2207.02489/) (2022)

## Built on by (most cited first, 2 of 2)

- [An Evaluation Framework for Network IDS/IPS Datasets: Leveraging MITRE ATT&CK and Industry Relevance Metrics](https://mlsanity.com/paper/2511.12743/) (2025)
- [GMA-SAWGAN-GP: A Novel Data Generative Framework to Enhance IDS Detection Performance](https://mlsanity.com/paper/2603.28838/) (2026)

## Abstract (arXiv)

Our paper provides empirical comparisons between recent IDSs to provide an objective comparison between them to help users choose the most appropriate solution based on their requirements. Our results show that no one solution is the best, but is dependent on external variables such as the types of attacks, complexity, and network environment in the dataset. For example, BoT_IoT and Stratosphere IoT datasets both capture IoT-related attacks, but the deep neural network performed the best when tested using the BoT_IoT dataset while HELAD performed the best when tested using the Stratosphere IoT dataset. So although we found that a deep neural network solution had the highest average F1 scores on tested datasets, it is not always the best-performing one. We further discuss difficulties in using IDS from literature and project repositories, which complicated drawing definitive conclusions regarding IDS selection.