# How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection

arXiv: https://arxiv.org/abs/2608.01454 · HTML: https://mlsanity.com/paper/2608.01454/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 65 references that are in the map.
- First posted on arXiv 2 Aug 2026.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [LogBERT: Log Anomaly Detection via BERT](https://mlsanity.com/paper/2103.04475/) (2021)
- [Dos and Don'ts of Machine Learning in Computer Security](https://mlsanity.com/paper/2010.09470/) (2020)
- [Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging](https://mlsanity.com/paper/1707.09861/) (2017)
- [E-GraphSAGE: A Graph Neural Network based Intrusion Detection System for IoT](https://mlsanity.com/paper/2103.16329/) (2021)
- [UNICORN: Runtime Provenance-Based Detector for Advanced Persistent Threats](https://mlsanity.com/paper/2001.01525/) (2020)
- [HOLMES: Real-time APT Detection through Correlation of Suspicious Information Flows](https://mlsanity.com/paper/1810.01594/) (2018)
- [Kairos: Practical Intrusion Detection and Investigation using Whole-system Provenance](https://mlsanity.com/paper/2308.05034/) (2023)
- [SLEUTH: Real-time Attack Scenario Reconstruction from COTS Audit Data](https://mlsanity.com/paper/1801.02062/) (2018)
- [threaTrace: Detecting and Tracing Host-based Threats in Node Level Through Provenance Graph Learning](https://mlsanity.com/paper/2111.04333/) (2021)
- [MAGIC: Detecting Advanced Persistent Threats via Masked Graph Representation Learning](https://mlsanity.com/paper/2310.09831/) (2023)
- [Practical Whole-System Provenance Capture](https://mlsanity.com/paper/1711.05296/) (2017)
- [NODLINK: An Online System for Fine-Grained APT Attack Detection and Investigation](https://mlsanity.com/paper/2311.02331/) (2023)
- [SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detection](https://mlsanity.com/paper/2305.00550/) (2023)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time (Extended Version)](https://mlsanity.com/paper/2402.01359/) (2024)
- [Are we there yet? An Industrial Viewpoint on Provenance-based Endpoint Detection and Response Tools](https://mlsanity.com/paper/2307.08349/) (2023)
- [ATLASv2: ATLAS Attack Engagements, Version 2](https://mlsanity.com/paper/2401.01341/) (2023)
- [Self-Supervised Learning of Graph Representations for Network Intrusion Detection](https://mlsanity.com/paper/2509.16625/) (2025)

## Abstract (arXiv)

Provenance-based intrusion detection systems (PIDS) frequently report strong performance, but the conclusions drawn from these results can be highly sensitive to benchmarking choices and evaluation protocols. We investigate this dependency by re-evaluating representative PIDS on public datasets that meet our audit, labeling, and calibration requirements. Focusing primarily on the audited DARPA TC E3 datasets, we apply a unified protocol with temporally separated test periods and validation-only checkpoint selection and threshold calibration, and ask which architectural claims are empirically supported. We find that alerting success and investigation utility can diverge sharply, as several systems surface attacks without providing enough process-level context to support forensic investigation. Across the four primary datasets, a simple allowlist built from executable names and paths observed during training matches or exceeds the selected learned baselines on key operating-point metrics, showing that comparable performance on these metrics is achievable using lexical novelty alone. Quantifying semantic signal quality through feature completeness and field entropy helps explain why several audited E3 datasets support alerting performance without reliably separating model architectures. Theia, however, provides the richest semantic signal and is the only primary dataset on which our reference model, Theseus, clearly improves both ranking and node-level recovery over the other learned systems. Overall, these findings reinforce the importance of interpreting architectural claims in PIDS together with the benchmark properties and evaluation protocol that produced them.