# An Assessment of ChatGPT on Log Data arXiv: https://arxiv.org/abs/2309.07938 ยท HTML: https://mlsanity.com/paper/2309.07938/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 95 references that are in the map. - First posted on arXiv 14 Sep 2023. ## Builds on (most cited first, 28 of 28) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018) - [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021) - [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022) - [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022) - [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023) - [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022) - [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022) - [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022) - [Black-Box Tuning for Language-Model-as-a-Service](https://mlsanity.com/paper/2201.03514/) (2022) - [Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness](https://mlsanity.com/paper/2304.11633/) (2023) - [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023) - [Predictive Business Process Monitoring with LSTM Neural Networks](https://mlsanity.com/paper/1612.02130/) (2016) - [GPTEval: A Survey on Assessments of ChatGPT and GPT-4](https://mlsanity.com/paper/2308.12488/) (2023) - [Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics](https://mlsanity.com/paper/2008.06448/) (2020) - [Predicting Process Behaviour using Deep Learning](https://mlsanity.com/paper/1612.04600/) (2016) - [Log-based Anomaly Detection Without Log Parsing](https://mlsanity.com/paper/2108.01955/) (2021) - [Tools and Benchmarks for Automated Log Parsing](https://mlsanity.com/paper/1811.03509/) (2018) - [ChatGPT: A Meta-Analysis after 2.5 Months](https://mlsanity.com/paper/2302.13795/) (2023) - [Survey and cross-benchmark comparison of remaining time prediction methods in business process monitoring](https://mlsanity.com/paper/1805.02896/) (2018) - [Predictive Monitoring of Business Processes](https://mlsanity.com/paper/1312.4874/) (2013) - [Log Parsing with Prompt-based Few-shot Learning](https://mlsanity.com/paper/2302.07435/) (2023) - [UniParser: A Unified Log Parser for Heterogeneous Log Data](https://mlsanity.com/paper/2202.06569/) (2022) - [A study on Prompt Design, Advantages and Limitations of ChatGPT for Deep Learning Program Repair](https://mlsanity.com/paper/2304.08191/) (2023) - [Detection of Early-Stage Enterprise Infection by Mining Large-Scale Log Data](https://mlsanity.com/paper/1411.5005/) (2014) - [ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks](https://mlsanity.com/paper/2305.16837/) (2023) - [Summarizing Unstructured Logs in Online Services](https://mlsanity.com/paper/2012.08938/) (2020) - [LogAI: A Library for Log Analytics and Intelligence](https://mlsanity.com/paper/2301.13415/) (2023) ## Built on by (most cited first, 12 of 12) - [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023) - [LogParser-LLM: Advancing Efficient Log Parsing with Large Language Models](https://mlsanity.com/paper/2408.13727/) (2024) - [RAGLog: Log Anomaly Detection using Retrieval Augmented Generation](https://mlsanity.com/paper/2311.05261/) (2023) - [A Survey of AIOps in the Era of Large Language Models](https://mlsanity.com/paper/2507.12472/) (2025) - [A Comparative Study on Large Language Models for Log Parsing](https://mlsanity.com/paper/2409.02474/) (2024) - [Mapping the Landscape of Generative AI in Network Monitoring and Management](https://mlsanity.com/paper/2502.08576/) (2025) - [HELP: Hierarchical Embeddings-based Log Parsing](https://mlsanity.com/paper/2408.08300/) (2024) - [Enhancing Reasoning Capacity of SLM using Cognitive Enhancement](https://mlsanity.com/paper/2404.01135/) (2024) - [Disrupting Test Development with AI Assistants](https://mlsanity.com/paper/2411.02328/) (2024) - [REFLEX: Reference-Free Evaluation of Log Summarization via Large Language Model Judgment](https://mlsanity.com/paper/2511.07458/) (2025) - [LogRouter: Adaptive Two-Level LLM Routing for Log Question Answering in Big Data Systems](https://mlsanity.com/paper/2605.18015/) (2026) - [Just Testing, Move Along: Evasion of LLM-based System Log Interpretation by Prompt Injection](https://mlsanity.com/paper/2607.24174/) (2026) ## Abstract (arXiv) Recent development of large language models (LLMs), such as ChatGPT has been widely applied to a wide range of software engineering tasks. Many papers have reported their analysis on the potential advantages and limitations of ChatGPT for writing code, summarization, text generation, etc. However, the analysis of the current state of ChatGPT for log processing has received little attention. Logs generated by large-scale software systems are complex and hard to understand. Despite their complexity, they provide crucial information for subject matter experts to understand the system status and diagnose problems of the systems. In this paper, we investigate the current capabilities of ChatGPT to perform several interesting tasks on log data, while also trying to identify its main shortcomings. Our findings show that the performance of the current version of ChatGPT for log processing is limited, with a lack of consistency in responses and scalability issues. We also outline our views on how we perceive the role of LLMs in the log processing discipline and possible next steps to improve the current capabilities of ChatGPT and the future LLMs in this area. We believe our work can contribute to future academic research to address the identified issues.