# Benchmarking the Benchmark -- Analysis of Synthetic NIDS Datasets

arXiv: https://arxiv.org/abs/2104.09029 · HTML: https://mlsanity.com/paper/2104.09029/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 37 references that are in the map.
- First posted on arXiv 19 Apr 2021.

## Builds on (most cited first, 3 of 3)

- [On Wasserstein Two Sample Testing and Related Families of Nonparametric Tests](https://mlsanity.com/paper/1509.02237/) (2015)
- [A Survey of Network-based Intrusion Detection Data Sets](https://mlsanity.com/paper/1903.02460/) (2019)
- [NetFlow Datasets for Machine Learning-based Network Intrusion Detection Systems](https://mlsanity.com/paper/2011.09144/) (2020)

## Built on by (most cited first, 5 of 5)

- [Towards Explainable Network Intrusion Detection using Large Language Models](https://mlsanity.com/paper/2408.04342/) (2024)
- [From Zero-Shot Machine Learning to Zero-Day Attack Detection](https://mlsanity.com/paper/2109.14868/) (2021)
- [On Generalisability of Machine Learning-based Network Intrusion Detection Systems](https://mlsanity.com/paper/2205.04112/) (2022)
- [DI-NIDS: Domain Invariant Network Intrusion Detection System](https://mlsanity.com/paper/2210.08252/) (2022)
- [A Cyber Threat Intelligence Sharing Scheme based on Federated Learning for Network Intrusion Detection](https://mlsanity.com/paper/2111.02791/) (2021)

## Abstract (arXiv)

Network Intrusion Detection Systems (NIDSs) are an increasingly important tool for the prevention and mitigation of cyber attacks. A number of labelled synthetic datasets generated have been generated and made publicly available by researchers, and they have become the benchmarks via which new ML-based NIDS classifiers are being evaluated. Recently published results show excellent classification performance with these datasets, increasingly approaching 100 percent performance across key evaluation metrics such as accuracy, F1 score, etc. Unfortunately, we have not yet seen these excellent academic research results translated into practical NIDS systems with such near-perfect performance. This motivated our research presented in this paper, where we analyse the statistical properties of the benign traffic in three of the more recent and relevant NIDS datasets, (CIC, UNSW, ...). As a comparison, we consider two datasets obtained from real-world production networks, one from a university network and one from a medium size Internet Service Provider (ISP). Our results show that the two real-world datasets are quite similar among themselves in regards to most of the considered statistical features. Equally, the three synthetic datasets are also relatively similar within their group. However, and most importantly, our results show a distinct difference of most of the considered statistical features between the three synthetic datasets and the two real-world datasets. Since ML relies on the basic assumption of training and test datasets being sampled from the same distribution, this raises the question of how well the performance results of ML-classifiers trained on the considered synthetic datasets can translate and generalise to real-world networks. We believe this is an interesting and relevant question which provides motivation for further research in this space.