# LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing

arXiv: https://arxiv.org/abs/2404.18001 · HTML: https://mlsanity.com/paper/2404.18001/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 75 references that are in the map.
- First posted on arXiv 27 Apr 2024.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation](https://mlsanity.com/paper/2305.16938/) (2023)
- [A Survey on Automated Log Analysis for Reliability Engineering](https://mlsanity.com/paper/2009.07237/) (2020)
- [Diversity in Machine Learning](https://mlsanity.com/paper/1807.01477/) (2018)
- [Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference](https://mlsanity.com/paper/2303.04673/) (2023)
- [UniParser: A Unified Log Parser for Heterogeneous Log Data](https://mlsanity.com/paper/2202.06569/) (2022)
- [Self-Supervised Log Parsing](https://mlsanity.com/paper/2003.07905/) (2020)
- [Did We Miss Something Important? Studying and Exploring Variable-Aware Log Abstraction](https://mlsanity.com/paper/2304.11391/) (2023)
- [Stability of Topic Modeling via Matrix Factorization](https://mlsanity.com/paper/1702.07186/) (2017)
- [Do Language Models Perform Generalizable Commonsense Inference?](https://mlsanity.com/paper/2106.11533/) (2021)

## Built on by (most cited first, 30 of 32)

- [TRAIL: Trace Reasoning and Agentic Issue Localization](https://mlsanity.com/paper/2505.08638/) (2025)
- [A Survey of AIOps in the Era of Large Language Models](https://mlsanity.com/paper/2507.12472/) (2025)
- [FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights](https://mlsanity.com/paper/2602.02905/) (2026)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [Adapting Large Language Models to Log Analysis with Interpretable Domain Knowledge](https://mlsanity.com/paper/2412.01377/) (2024)
- [VulScribeR: Exploring RAG-based Vulnerability Augmentation with LLMs](https://mlsanity.com/paper/2408.04125/) (2024)
- [Retrieval-Augmented Test Generation: How Far Are We?](https://mlsanity.com/paper/2409.12682/) (2024)
- [On the Effectiveness of Large Language Models in Writing Alloy Formulas](https://mlsanity.com/paper/2502.15441/) (2025)
- [The Multi-Agent Fault Localization System Based on Monte Carlo Tree Search Approach](https://mlsanity.com/paper/2507.22800/) (2025)
- [A Comparative Study on Large Language Models for Log Parsing](https://mlsanity.com/paper/2409.02474/) (2024)
- [Generative AI for Self-Adaptive Systems: State of the Art and Research Roadmap](https://mlsanity.com/paper/2512.04680/) (2025)
- [Log Parsing using LLMs with Self-Generated In-Context Learning and Self-Correction](https://mlsanity.com/paper/2406.03376/) (2024)
- [AdaptiveLog: An Adaptive Log Analysis Framework with the Collaboration of Large and Small Language Model](https://mlsanity.com/paper/2501.11031/) (2025)
- [Evaluating the Effectiveness of LLMs in Fixing Maintainability Issues in Real-World Projects](https://mlsanity.com/paper/2502.02368/) (2025)
- [Inducing Personality in LLM-Based Honeypot Agents: Measuring the Effect on Human-Like Agenda Generation](https://mlsanity.com/paper/2503.19752/) (2025)
- [LibreLog: Accurate and Efficient Unsupervised Log Parsing Using Open-Source Large Language Models](https://mlsanity.com/paper/2408.01585/) (2024)
- [System Log Parsing with Large Language Models: A Review](https://mlsanity.com/paper/2504.04877/) (2025)
- [Towards Practical Defect-Focused Automated Code Review](https://mlsanity.com/paper/2505.17928/) (2025)
- [LLMLogAnalyzer: A Clustering-Based Log Analysis Chatbot using Large Language Models](https://mlsanity.com/paper/2510.24031/) (2025)
- [LUK: Empowering Log Understanding with Expert Knowledge from Large Language Models](https://mlsanity.com/paper/2409.01909/) (2024)
- [OBsmith: LLM-Powered JavaScript Obfuscator Testing](https://mlsanity.com/paper/2510.10066/) (2025)
- [Benchmarking and Exploring the Capabilities of LLMs for Attack Investigations](https://mlsanity.com/paper/2606.10281/) (2026)
- [Chatting with Logs: An exploratory study on Finetuning LLMs for LogQL](https://mlsanity.com/paper/2412.03612/) (2024)
- [A Feature Engineering Approach for Business Impact-Oriented Failure Detection in Distributed Instant Payment Systems](https://mlsanity.com/paper/2510.21710/) (2025)
- [Intent-Driven Storage Systems: From Low-Level Tuning to High-Level Understanding](https://mlsanity.com/paper/2510.15917/) (2025)
- [KRONE: Scalable LLM-Augmented Log Anomaly Detection via Hierarchical Abstraction](https://mlsanity.com/paper/2602.07303/) (2026)
- [On Sequence-to-Sequence Models for Automated Log Parsing](https://mlsanity.com/paper/2602.07698/) (2026)
- [An End-to-End Framework for Functionality-Embedded Provenance Graph Construction and Threat Interpretation](https://mlsanity.com/paper/2603.17100/) (2026)
- [Tabular PDF Information Extraction with Local LLMs and Layout-Aware Parsing: A Reliability Evaluation](https://mlsanity.com/paper/2604.00003/) (2026)
- [Instruction-Tuned LLMs for Parsing and Mining Unstructured Logs on Leadership HPC Systems](https://mlsanity.com/paper/2604.05168/) (2026)

## Abstract (arXiv)

Logs are important in modern software development with runtime information. Log parsing is the first step in many log-based analyses, that involve extracting structured information from unstructured log data. Traditional log parsers face challenges in accurately parsing logs due to the diversity of log formats, which directly impacts the performance of downstream log-analysis tasks. In this paper, we explore the potential of using Large Language Models (LLMs) for log parsing and propose LLMParser, an LLM-based log parser based on generative LLMs and few-shot tuning. We leverage four LLMs, Flan-T5-small, Flan-T5-base, LLaMA-7B, and ChatGLM-6B in LLMParsers. Our evaluation of 16 open-source systems shows that LLMParser achieves statistically significantly higher parsing accuracy than state-of-the-art parsers (a 96% average parsing accuracy). We further conduct a comprehensive empirical analysis on the effect of training size, model size, and pre-training LLM on log parsing accuracy. We find that smaller LLMs may be more effective than more complex LLMs; for instance where Flan-T5-base achieves comparable results as LLaMA-7B with a shorter inference time. We also find that using LLMs pre-trained using logs from other systems does not always improve parsing accuracy. While using pre-trained Flan-T5-base shows an improvement in accuracy, pre-trained LLaMA results in a decrease (decrease by almost 55% in group accuracy). In short, our study provides empirical evidence for using LLMs for log parsing and highlights the limitations and future research direction of LLM-based log parsers.