# Protocol-Sensitive Evaluation of Log Anomaly Detection: Component Costs and Target-Access Sensitivity on HDFS and BGL

arXiv: https://arxiv.org/abs/2610.05807 · HTML: https://mlsanity.com/paper/2610.05807/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 13 of 13)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [LogBERT: Log Anomaly Detection via BERT](https://mlsanity.com/paper/2103.04475/) (2021)
- [Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics](https://mlsanity.com/paper/2008.06448/) (2020)
- [Log-based Anomaly Detection with Deep Learning: How Far Are We?](https://mlsanity.com/paper/2202.04301/) (2022)
- [Deep Learning for Anomaly Detection in Log Data: A Survey](https://mlsanity.com/paper/2207.03820/) (2022)
- [LogLLM: Log-based Anomaly Detection Using Large Language Models](https://mlsanity.com/paper/2411.08561/) (2024)
- [SemParser: A Semantic Parser for Log Analysis](https://mlsanity.com/paper/2112.12636/) (2021)
- [A Critical Review of Common Log Data Sets Used for Evaluation of Sequence-based Anomaly Detection Techniques](https://mlsanity.com/paper/2309.02854/) (2023)
- [LogSD: Detecting Anomalies from System Logs through Self-supervised Learning and Frequency-based Masking](https://mlsanity.com/paper/2404.11294/) (2024)
- [Impact of Log Parsing on Deep Learning-Based Anomaly Detection](https://mlsanity.com/paper/2305.15897/) (2023)
- [A Comprehensive Study of Machine Learning Techniques for Log-Based Anomaly Detection](https://mlsanity.com/paper/2307.16714/) (2023)
- [CAShift: Benchmarking Log-Based Cloud Attack Detection under Normality Shift](https://mlsanity.com/paper/2504.09115/) (2025)

## Abstract (arXiv)

Protocol choices can change the conclusions drawn from log anomaly detection benchmarks even when detector settings are fixed. We present a joint empirical study of split construction, representation visibility, and component costs using six fixed count, sequence, and semantic configurations on Hadoop Distributed File System (HDFS) and Blue Gene/L (BGL) logs. Random splits place several configurations near the average-precision ceiling, whereas group-disjoint HDFS and chronological BGL evaluation produce lower scores and different observed orderings. At a fixed BGL cutoff, parser choice spans 0.124 in semantic XGBoost mean average precision while preserving its lead over count XGBoost; the earliest rolling period reverses that ordering. A two-factor cross-system ablation contrasts source-only representations with offline transductive access to unlabeled target templates through the representation corpus and inverse document frequency: HDFS-to-BGL mean average precision moves from 0.191 with source-only access to 0.325 with union-corpus, target-IDF access, and the intermediate conditions reveal direction-dependent interactions in average precision and retrieval at fixed review budgets. Component-level profiling separates parsing and representation costs from classifier training, prediction, and storage. Together, these findings connect detector comparisons to the test population, preprocessing state, visible information, and measured pipeline stages, and identify the protocol fields needed alongside a score to support interpretable comparisons of log anomaly detection accuracy and resource use.