# Log Parsing Evaluation in the Era of Modern Software Systems

arXiv: https://arxiv.org/abs/2308.09003 · HTML: https://mlsanity.com/paper/2308.09003/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Aug 2023.

## Built on by (most cited first, 6 of 6)

- [A Comparative Study on Large Language Models for Log Parsing](https://mlsanity.com/paper/2409.02474/) (2024)
- [Log Parsing using LLMs with Self-Generated In-Context Learning and Self-Correction](https://mlsanity.com/paper/2406.03376/) (2024)
- [System Log Parsing with Large Language Models: A Review](https://mlsanity.com/paper/2504.04877/) (2025)
- [HELP: Hierarchical Embeddings-based Log Parsing](https://mlsanity.com/paper/2408.08300/) (2024)
- [A Word is Worth 4-bit: Efficient Log Parsing with Binary Coded Decimal Recognition](https://mlsanity.com/paper/2506.01147/) (2025)
- [FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection](https://mlsanity.com/paper/2605.22779/) (2026)

## Abstract (arXiv)

Due to the complexity and size of modern software systems, the amount of logs generated is tremendous. Hence, it is infeasible to manually investigate these data in a reasonable time, thereby requiring automating log analysis to derive insights about the functioning of the systems. Motivated by an industry use-case, we zoom-in on one integral part of automated log analysis, log parsing, which is the prerequisite to deriving any insights from logs. Our investigation reveals problematic aspects within the log parsing field, particularly its inefficiency in handling heterogeneous real-world logs. We show this by assessing the 14 most-recognized log parsing approaches in the literature using (i) nine publicly available datasets, (ii) one dataset comprised of combined publicly available data, and (iii) one dataset generated within the infrastructure of a large bank. Subsequently, toward improving log parsing robustness in real-world production scenarios, we propose a tool, Logchimera, that enables estimating log parsing performance in industry contexts through generating synthetic log data that resemble industry logs. Our contributions serve as a foundation to consolidate past research efforts, facilitate future research advancements, and establish a strong link between research and industry log parsing.