# An Evaluation Framework for Network IDS/IPS Datasets: Leveraging MITRE ATT&CK and Industry Relevance Metrics

arXiv: https://arxiv.org/abs/2511.12743 · HTML: https://mlsanity.com/paper/2511.12743/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 74 references that are in the map.
- First posted on arXiv 16 Nov 2025.

## Builds on (most cited first, 16 of 16)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [A Survey of Network-based Intrusion Detection Data Sets](https://mlsanity.com/paper/1903.02460/) (2019)
- [Towards a Standard Feature Set for Network Intrusion Detection System Datasets](https://mlsanity.com/paper/2101.11315/) (2021)
- [Ethical Considerations for Responsible Data Curation](https://mlsanity.com/paper/2302.03629/) (2023)
- [Machine learning-based network intrusion detection for big and imbalanced data using oversampling, stacking feature embedding and feature extraction](https://mlsanity.com/paper/2401.12262/) (2024)
- [A Dependable Hybrid Machine Learning Model for Network Intrusion Detection](https://mlsanity.com/paper/2212.04546/) (2022)
- [MITRE ATT&CK: State of the Art and Way Forward](https://mlsanity.com/paper/2308.14016/) (2023)
- [Network Intrusion Datasets: A Survey, Limitations, and Recommendations](https://mlsanity.com/paper/2502.06688/) (2025)
- [FedDef: Defense Against Gradient Leakage in Federated Learning-based Network Intrusion Detection Systems](https://mlsanity.com/paper/2210.04052/) (2022)
- [A Robust Comparison of the KDDCup99 and NSL-KDD IoT Network Intrusion Detection Datasets Through Various Machine Learning Algorithms](https://mlsanity.com/paper/1912.13204/) (2019)
- [Bridging the gap to real-world for network intrusion detection systems with data-centric approach](https://mlsanity.com/paper/2110.13655/) (2021)
- [Expectations Versus Reality: Evaluating Intrusion Detection Systems in Practice](https://mlsanity.com/paper/2403.17458/) (2024)
- [Emerging Trends of Recently Published Datasets for Intrusion Detection Systems (IDS): A Survey](https://mlsanity.com/paper/2110.00773/) (2021)
- [ForensiBlock: A Provenance-Driven Blockchain Framework for Data Forensics and Auditability](https://mlsanity.com/paper/2308.03927/) (2023)
- [A Secured Triad of IoT, Machine Learning, and Blockchain for Crop Forecasting in Agriculture](https://mlsanity.com/paper/2505.01196/) (2025)

## Built on by (most cited first, 1 of 1)

- [Privacy, Robustness, and Fairness Trade-offs in Federated Intrusion Detection: Geometric Indistinguishability at the Aggregation Interface](https://mlsanity.com/paper/2609.03420/) (2026)

## Abstract (arXiv)

The performance of Machine Learning (ML) and Deep Learning (DL)-based Intrusion Detection and Prevention Systems (IDS/IPS) is critically dependent on the relevance and quality of the datasets used for training and evaluation. However, current AI model evaluation practices for developing IDS/IPS focus predominantly on accuracy metrics, often overlooking whether datasets represent industry-specific threats. To address this gap, we introduce a novel multi-dimensional framework that integrates the MITRE ATT&CK knowledge base for threat intelligence and employs five complementary metrics that together provide a comprehensive assessment of dataset suitability. Methodologically, this framework combines threat intelligence, natural language processing, and quantitative analysis to assess the suitability of datasets for specific industry contexts. Applying this framework to nine publicly available IDS/IPS datasets reveals significant gaps in threat coverage, particularly in the healthcare, energy, and financial sectors. In particular, recent datasets (e.g., CIC-IoMT, CIC-UNSW-NB15) align better with sector-specific threats, whereas others, like CICIoV-24, underperform despite their recency. Our findings provide a standardized, interpretable approach for selecting datasets aligned with sector-specific operational requirements, ultimately enhancing the real-world effectiveness of AI-driven IDS/IPS deployments. The efficiency and practicality of the framework are validated through deployment in a real-world case study, underscoring its capacity to inform dataset selection and enhance the effectiveness of AI-driven IDS/IPS in operational environments.