# Tools and Benchmarks for Automated Log Parsing

arXiv: https://arxiv.org/abs/1811.03509 · HTML: https://mlsanity.com/paper/1811.03509/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2018.

## Built on by (most cited first, 30 of 56)

- [Log-based Anomaly Detection Without Log Parsing](https://mlsanity.com/paper/2108.01955/) (2021)
- [Log-based Anomaly Detection with Deep Learning: How Far Are We?](https://mlsanity.com/paper/2202.04301/) (2022)
- [Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection](https://mlsanity.com/paper/2107.05908/) (2021)
- [Automatic Programming: Large Language Models and Beyond](https://mlsanity.com/paper/2405.02213/) (2024)
- [Deep Learning for Anomaly Detection in Log Data: A Survey](https://mlsanity.com/paper/2207.03820/) (2022)
- [LogLLM: Log-based Anomaly Detection Using Large Language Models](https://mlsanity.com/paper/2411.08561/) (2024)
- [Building AI Agents for Autonomous Clouds: Challenges and Design Principles](https://mlsanity.com/paper/2407.12165/) (2024)
- [Log Parsing: How Far Can ChatGPT Go?](https://mlsanity.com/paper/2306.01590/) (2023)
- [OptLLM: Optimal Assignment of Queries to Large Language Models](https://mlsanity.com/paper/2405.15130/) (2024)
- [LogLM: From Task-based to Instruction-based Automated Log Analysis](https://mlsanity.com/paper/2410.09352/) (2024)
- [Robust and Transferable Anomaly Detection in Log Data using Pre-Trained Language Models](https://mlsanity.com/paper/2102.11570/) (2021)
- [An Assessment of ChatGPT on Log Data](https://mlsanity.com/paper/2309.07938/) (2023)
- [UniLog: Deploy One Model and Specialize it for All Log Analysis Tasks](https://mlsanity.com/paper/2112.03159/) (2021)
- [LogEval: A Comprehensive Benchmark Suite for Large Language Models In Log Analysis](https://mlsanity.com/paper/2407.01896/) (2024)
- [Heterogeneous Anomaly Detection for Software Systems via Semi-supervised Cross-modal Attention](https://mlsanity.com/paper/2302.06914/) (2023)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [LogGD:Detecting Anomalies from System Logs by Graph Neural Networks](https://mlsanity.com/paper/2209.07869/) (2022)
- [Graph Neural Networks based Log Anomaly Detection and Explanation](https://mlsanity.com/paper/2307.00527/) (2023)
- [Adapting Large Language Models to Log Analysis with Interpretable Domain Knowledge](https://mlsanity.com/paper/2412.01377/) (2024)
- [Log Anomaly Detection with Large Language Models via Knowledge-Enriched Fusion](https://mlsanity.com/paper/2512.11997/) (2025)
- [OneLog: Towards End-to-End Training in Software Log Anomaly Detection](https://mlsanity.com/paper/2104.07324/) (2021)
- [A Comprehensive Survey of Logging in Software: From Logging Statements Automation to Log Mining and Analysis](https://mlsanity.com/paper/2110.12489/) (2021)
- [On the Effectiveness of Log Representation for Log-based Anomaly Detection](https://mlsanity.com/paper/2308.08736/) (2023)
- [Learning Representations on Logs for AIOps](https://mlsanity.com/paper/2308.11526/) (2023)
- [Demystifying and Extracting Fault-indicating Information from Logs for Failure Diagnosis](https://mlsanity.com/paper/2409.13561/) (2024)
- [A Comparative Study on Large Language Models for Log Parsing](https://mlsanity.com/paper/2409.02474/) (2024)
- [Multi-Source Anomaly Detection in Distributed IT Systems](https://mlsanity.com/paper/2101.04977/) (2021)
- [LogELECTRA: Self-supervised Anomaly Detection for Unstructured Logs](https://mlsanity.com/paper/2402.10397/) (2024)
- [Log Parsing using LLMs with Self-Generated In-Context Learning and Self-Correction](https://mlsanity.com/paper/2406.03376/) (2024)
- [Feature Selection for Fault Detection and Prediction based on Event Log Analysis](https://mlsanity.com/paper/2208.09440/) (2022)

## Abstract (arXiv)

Logs are imperative in the development and maintenance process of many software systems. They record detailed runtime information that allows developers and support engineers to monitor their systems and dissect anomalous behaviors and errors. The increasing scale and complexity of modern software systems, however, make the volume of logs explodes. In many cases, the traditional way of manual log inspection becomes impractical. Many recent studies, as well as industrial tools, resort to powerful text search and machine learning-based analytics solutions. Due to the unstructured nature of logs, a first crucial step is to parse log messages into structured data for subsequent analysis. In recent years, automated log parsing has been widely studied in both academia and industry, producing a series of log parsers by different techniques. To better understand the characteristics of these log parsers, in this paper, we present a comprehensive evaluation study on automated log parsing and further release the tools and benchmarks for easy reuse. More specifically, we evaluate 13 log parsers on a total of 16 log datasets spanning distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software. We report the benchmarking results in terms of accuracy, robustness, and efficiency, which are of practical importance when deploying automated log parsing in production. We also share the success stories and lessons learned in an industrial application at Huawei. We believe that our work could serve as the basis and provide valuable guidance to future research and deployment of automated log parsing.