# A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We?

arXiv: https://arxiv.org/abs/2308.10828 · HTML: https://mlsanity.com/paper/2308.10828/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2023.

## Built on by (most cited first, 26 of 26)

- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [Log Anomaly Detection with Large Language Models via Knowledge-Enriched Fusion](https://mlsanity.com/paper/2512.11997/) (2025)
- [Demystifying and Extracting Fault-indicating Information from Logs for Failure Diagnosis](https://mlsanity.com/paper/2409.13561/) (2024)
- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [A Comparative Study on Large Language Models for Log Parsing](https://mlsanity.com/paper/2409.02474/) (2024)
- [Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality](https://mlsanity.com/paper/2503.05860/) (2025)
- [System Log Parsing with Large Language Models: A Review](https://mlsanity.com/paper/2504.04877/) (2025)
- [SDLog: A Deep Learning Framework for Detecting Sensitive Information in Software Logs](https://mlsanity.com/paper/2505.14976/) (2025)
- [SchemaCoder: Automatic Log Schema Extraction Coder with Residual Q-Tree Boosting](https://mlsanity.com/paper/2508.18554/) (2025)
- [Bio-inspired Agentic Self-healing Framework for Resilient Distributed Computing Continuum Systems](https://mlsanity.com/paper/2601.00339/) (2026)
- [NeSy-Edge: Neuro-Symbolic Trustworthy Self-Healing in the Computing Continuum](https://mlsanity.com/paper/2603.21145/) (2026)
- [Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMs](https://mlsanity.com/paper/2604.20211/) (2026)
- [LogBabylon: A Unified Framework for Cross-Log File Integration and Analysis](https://mlsanity.com/paper/2412.12364/) (2024)
- [Multi-Agent Systems for Dataset Adaptation in Software Engineering: Capabilities, Limitations, and Future Directions](https://mlsanity.com/paper/2511.21380/) (2025)
- [A Word is Worth 4-bit: Efficient Log Parsing with Binary Coded Decimal Recognition](https://mlsanity.com/paper/2506.01147/) (2025)
- [FusionLog: Cross-System Log-based Anomaly Detection via Fusion of General and Proprietary Knowledge](https://mlsanity.com/paper/2511.05878/) (2025)
- [REFLEX: Reference-Free Evaluation of Log Summarization via Large Language Model Judgment](https://mlsanity.com/paper/2511.07458/) (2025)
- [Scalable and Efficient Large-Scale Log Analysis with LLMs: An IT Software Support Case Study](https://mlsanity.com/paper/2511.14803/) (2025)
- [MicLog: Towards Accurate and Efficient LLM-based Log Parsing via Progressive Meta In-Context Learning](https://mlsanity.com/paper/2601.07005/) (2026)
- [SCOPE: Tree-based Self-Correcting Online Log Parsing via Syntactic-Semantic Collaboration](https://mlsanity.com/paper/2603.27247/) (2026)
- [Lossless Prompt Compression via Dictionary-Encoding and In-Context Learning: Enabling Cost-Effective LLM Analysis of Repetitive Data](https://mlsanity.com/paper/2604.13066/) (2026)
- [Continuous Behavioral Authentication via Multi-Expert BERT Log Analysis for Secure Data Sharing](https://mlsanity.com/paper/2606.21900/) (2026)
- [Evaluating Agentic Code Repair Capabilities in Distributed Systems](https://mlsanity.com/paper/2608.14863/) (2026)
- [EdgeCraft: Automated Model Crafting for Edge IoT](https://mlsanity.com/paper/2609.35167/) (2026)
- [SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data](https://mlsanity.com/paper/2610.16412/) (2026)

## Abstract (arXiv)

Log data have facilitated various tasks of software development and maintenance, such as testing, debugging and diagnosing. Due to the unstructured nature of logs, log parsing is typically required to transform log messages into structured data for automated log analysis. Given the abundance of log parsers that employ various techniques, evaluating these tools to comprehend their characteristics and performance becomes imperative. Loghub serves as a commonly used dataset for benchmarking log parsers, but it suffers from limited scale and representativeness, posing significant challenges for studies to comprehensively evaluate existing log parsers or develop new methods. This limitation is particularly pronounced when assessing these log parsers for production use. To address these limitations, we provide a new collection of annotated log datasets, denoted Loghub-2.0, which can better reflect the characteristics of log data in real-world software systems. Loghub-2.0 comprises 14 datasets with an average of 3.6 million log lines in each dataset. Based on Loghub-2.0, we conduct a thorough re-evaluation of 15 state-of-the-art log parsers in a more rigorous and practical setting. Particularly, we introduce a new evaluation metric to mitigate the sensitivity of existing metrics to imbalanced data distributions. We are also the first to investigate the granular performance of log parsers on logs that represent rare system events, offering in-depth details for software diagnosis. Accurately parsing such logs is essential, yet it remains a challenge. We believe this work could shed light on the evaluation and design of log parsers in practical settings, thereby facilitating their deployment in production systems.