# LogLead -- Fast and Integrated Log Loader, Enhancer, and Anomaly Detector

arXiv: https://arxiv.org/abs/2311.11809 · HTML: https://mlsanity.com/paper/2311.11809/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 20 Nov 2023.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics](https://mlsanity.com/paper/2008.06448/) (2020)
- [Log-based Anomaly Detection Without Log Parsing](https://mlsanity.com/paper/2108.01955/) (2021)
- [Tools and Benchmarks for Automated Log Parsing](https://mlsanity.com/paper/1811.03509/) (2018)
- [Deep Learning for Anomaly Detection in Log Data: A Survey](https://mlsanity.com/paper/2207.03820/) (2022)
- [LogShrink: Effective Log Compression by Leveraging Commonality and Variability of Log Data](https://mlsanity.com/paper/2309.09479/) (2023)
- [On the Effectiveness of Log Representation for Log-based Anomaly Detection](https://mlsanity.com/paper/2308.08736/) (2023)
- [Pinpointing Anomaly Events in Logs from Stability Testing -- N-Grams vs. Deep-Learning](https://mlsanity.com/paper/2202.09214/) (2022)

## Built on by (most cited first, 1 of 1)

- [Cross-System Categorization of Abnormal Traces in Microservice-Based Systems via Meta-Learning](https://mlsanity.com/paper/2403.18998/) (2024)

## Abstract (arXiv)

This paper introduces LogLead, a tool designed for efficient log analysis benchmarking. LogLead combines three essential steps in log processing: loading, enhancing, and anomaly detection. The tool leverages Polars, a high-speed DataFrame library. We currently have Loaders for eight systems that are publicly available (HDFS, Hadoop, BGL, Thunderbird, Spirit, Liberty, TrainTicket, and GC Webshop). We have multiple enhancers with three parsers (Drain, Spell, LenMa), Bert embedding creation and other log representation techniques like bag-of-words. LogLead integrates to five supervised and four unsupervised machine learning algorithms for anomaly detection from SKLearn. By integrating diverse datasets, log representation methods and anomaly detectors, LogLead facilitates comprehensive benchmarking in log analysis research. We show that log loading from raw file to dataframe is over 10x faster with LogLead compared to past solutions. We demonstrate roughly 2x improvement in Drain parsing speed by off-loading log message normalization to LogLead. Our brief benchmarking on HDFS indicates that log representations extending beyond the bag-of-words approach offer limited additional benefits. Tool URL: https://github.com/EvoTestOps/LogLead