# A Comparative Study on Large Language Models for Log Parsing

arXiv: https://arxiv.org/abs/2409.02474 · HTML: https://mlsanity.com/paper/2409.02474/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 4 Sep 2024.

## Builds on (most cited first, 17 of 17)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [LogBERT: Log Anomaly Detection via BERT](https://mlsanity.com/paper/2103.04475/) (2021)
- [Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics](https://mlsanity.com/paper/2008.06448/) (2020)
- [Log-based Anomaly Detection with Deep Learning: How Far Are We?](https://mlsanity.com/paper/2202.04301/) (2022)
- [Tools and Benchmarks for Automated Log Parsing](https://mlsanity.com/paper/1811.03509/) (2018)
- [Interpretable Online Log Analysis Using Large Language Models with Prompt Strategies](https://mlsanity.com/paper/2308.07610/) (2023)
- [LILAC: Log Parsing using LLMs with Adaptive Parsing Cache](https://mlsanity.com/paper/2310.01796/) (2023)
- [LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing](https://mlsanity.com/paper/2404.18001/) (2024)
- [LAnoBERT: System Log Anomaly Detection based on BERT Masked Language Model](https://mlsanity.com/paper/2111.09564/) (2021)
- [UniParser: A Unified Log Parser for Heterogeneous Log Data](https://mlsanity.com/paper/2202.06569/) (2022)
- [A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We?](https://mlsanity.com/paper/2308.10828/) (2023)
- [Self-Supervised Log Parsing](https://mlsanity.com/paper/2003.07905/) (2020)
- [Log Parsing: How Far Can ChatGPT Go?](https://mlsanity.com/paper/2306.01590/) (2023)
- [An Assessment of ChatGPT on Log Data](https://mlsanity.com/paper/2309.07938/) (2023)
- [Log Parsing Evaluation in the Era of Modern Software Systems](https://mlsanity.com/paper/2308.09003/) (2023)

## Built on by (most cited first, 5 of 5)

- [Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency](https://mlsanity.com/paper/2504.03360/) (2025)
- [AdaptiveLog: An Adaptive Log Analysis Framework with the Collaboration of Large and Small Language Model](https://mlsanity.com/paper/2501.11031/) (2025)
- [System Log Parsing with Large Language Models: A Review](https://mlsanity.com/paper/2504.04877/) (2025)
- [REFLEX: Reference-Free Evaluation of Log Summarization via Large Language Model Judgment](https://mlsanity.com/paper/2511.07458/) (2025)
- [Tabular PDF Information Extraction with Local LLMs and Layout-Aware Parsing: A Reliability Evaluation](https://mlsanity.com/paper/2604.00003/) (2026)

## Abstract (arXiv)

Background: Log messages provide valuable information about the status of software systems. This information is provided in an unstructured fashion and automated approaches are applied to extract relevant parameters. To ease this process, log parsing can be applied, which transforms log messages into structured log templates. Recent advances in language models have led to several studies that apply ChatGPT to the task of log parsing with promising results. However, the performance of other state-of-the-art large language models (LLMs) on the log parsing task remains unclear. Aims: In this study, we investigate the current capability of state-of-the-art LLMs to perform log parsing. Method: We select six recent LLMs, including both paid proprietary (GPT-3.5, Claude 2.1) and four free-to-use open models, and compare their performance on system logs obtained from a selection of mature open-source projects. We design two different prompting approaches and apply the LLMs on 1, 354 log templates across 16 different projects. We evaluate their effectiveness, in the number of correctly identified templates, and the syntactic similarity between the generated templates and the ground truth. Results: We found that free-to-use models are able to compete with paid models, with CodeLlama extracting 10% more log templates correctly than GPT-3.5. Moreover, we provide qualitative insights into the usability of language models (e.g., how easy it is to use their responses). Conclusions: Our results reveal that some of the smaller, free-to-use LLMs can considerably assist log parsing compared to their paid proprietary competitors, especially code-specialized models.