# On the Cross-Dataset Generalization of Machine Learning for Network Intrusion Detection

arXiv: https://arxiv.org/abs/2402.10974 · HTML: https://mlsanity.com/paper/2402.10974/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 74 references that are in the map.
- First posted on arXiv 15 Feb 2024.

## Builds on (most cited first, 9 of 9)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Kitsune: An Ensemble of Autoencoders for Online Network Intrusion Detection](https://mlsanity.com/paper/1802.09089/) (2018)
- [Towards a Standard Feature Set for Network Intrusion Detection System Datasets](https://mlsanity.com/paper/2101.11315/) (2021)
- [Federated Learning for Intrusion Detection System: Concepts, Challenges and Future Directions](https://mlsanity.com/paper/2106.09527/) (2021)
- [Building an Efficient Intrusion Detection System Based on Feature Selection and Ensemble Classifier](https://mlsanity.com/paper/1904.01352/) (2019)
- [Assessing Software Defection Prediction Performance: Why Using the Matthews Correlation Coefficient Matters](https://mlsanity.com/paper/2003.01182/) (2020)
- [Evaluating Standard Feature Sets Towards Increased Generalisability and Explainability of ML-based Network Intrusion Detection](https://mlsanity.com/paper/2104.07183/) (2021)
- [A Cyber Threat Intelligence Sharing Scheme based on Federated Learning for Network Intrusion Detection](https://mlsanity.com/paper/2111.02791/) (2021)
- [Generalizing intrusion detection for heterogeneous networks: A stacked-unsupervised federated learning approach](https://mlsanity.com/paper/2209.00721/) (2022)

## Built on by (most cited first, 4 of 4)

- [Cross-Domain Generalization Failure in Lightweight Intrusion Detection Models for IIoT Networks](https://mlsanity.com/paper/2607.00553/) (2026)
- [Machine Learning for Network Attacks Classification and Statistical Evaluation of Adversarial Learning Methodologies for Synthetic Data Generation](https://mlsanity.com/paper/2603.17717/) (2026)
- [MASH-Bench: Diagnosing Cross-Source Failure in Mass-Shooting Risk Classification](https://mlsanity.com/paper/2608.22460/) (2026)
- [A Three-Axis Stress Test of LLM vs Classical ML for Network Intrusion Detection under Distribution Shift and Adversarial Evasion](https://mlsanity.com/paper/2609.13511/) (2026)

## Abstract (arXiv)

Network Intrusion Detection Systems (NIDS) are a fundamental tool in cybersecurity. Their ability to generalize across diverse networks is a critical factor in their effectiveness and a prerequisite for real-world applications. In this study, we conduct a comprehensive analysis on the generalization of machine-learning-based NIDS through an extensive experimentation in a cross-dataset framework. We employ four machine learning classifiers and utilize four datasets acquired from different networks: CIC-IDS-2017, CSE-CIC-IDS2018, LycoS-IDS2017, and LycoS-Unicas-IDS2018. Notably, the last dataset is a novel contribution, where we apply corrections based on LycoS-IDS2017 to the well-known CSE-CIC-IDS2018 dataset. The results show nearly perfect classification performance when the models are trained and tested on the same dataset. However, when training and testing the models in a cross-dataset fashion, the classification accuracy is largely commensurate with random chance except for a few combinations of attacks and datasets. We employ data visualization techniques in order to provide valuable insights on the patterns in the data. Our analysis unveils the presence of anomalies in the data that directly hinder the classifiers capability to generalize the learned knowledge to new scenarios. This study enhances our comprehension of the generalization capabilities of machine-learning-based NIDS, highlighting the significance of acknowledging data heterogeneity.