mlsanityData to 7 Oct 2026

An Assessment of ChatGPT on Log Data

arXiv 2309.07938 ↗First posted 14 Sep 2023cs.SE cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Recent development of large language models (LLMs), such as ChatGPT has been widely applied to a wide range of software engineering tasks. Many papers have reported their analysis on the potential advantages and limitations of ChatGPT for writing code, summarization, text generation, etc. However, the analysis of the current state of ChatGPT for log processing has received little attention. Logs generated by large-scale software systems are complex and hard to understand. Despite their complexity, they provide crucial information for subject matter experts to understand the system status and diagnose problems of the systems. In this paper, we investigate the current capabilities of ChatGPT to perform several interesting tasks on log data, while also trying to identify its main shortcomings. Our findings show that the performance of the current version of ChatGPT for log processing is limited, with a lack of consistency in responses and scalability issues. We also outline our views on how we perceive the role of LLMs in the log processing discipline and possible next steps to improve the current capabilities of ChatGPT and the future LLMs in this area. We believe our work can contribute to future academic research to address the identified issues.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 28 of its 95 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

28 of 95 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  9. 2022UL2: Unifying Language Learning Paradigmscited by 292
  10. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  11. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  12. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
16 more
  1. 2016Predictive Business Process Monitoring with LSTM Neural Networkscited by 103
  2. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  3. 2020Loghub: A Large Collection of System Log Datasets for AI-driven Log Analyticscited by 68
  4. 2016Predicting Process Behaviour using Deep Learningcited by 59
  5. 2021Log-based Anomaly Detection Without Log Parsingcited by 59
  6. 2018Tools and Benchmarks for Automated Log Parsingcited by 56
  7. 2023ChatGPT: A Meta-Analysis after 2.5 Monthscited by 50
  8. 2018Survey and cross-benchmark comparison of remaining time prediction methods in business process monitoringcited by 43
  9. 2013Predictive Monitoring of Business Processescited by 37
  10. 2023Log Parsing with Prompt-based Few-shot Learningcited by 28
  11. 2023A study on Prompt Design, Advantages and Limitations of ChatGPT for Deep Learning Program Repaircited by 26
  12. 2022UniParser: A Unified Log Parser for Heterogeneous Log Datacited by 25
  13. 2014Detection of Early-Stage Enterprise Infection by Mining Large-Scale Log Datacited by 13
  14. 2023ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Taskscited by 12
  15. 2020Summarizing Unstructured Logs in Online Servicescited by 5
  16. 2023LogAI: A Library for Log Analytics and Intelligencecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 3; 2026: 1.

11 papers.

  1. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  2. 2024LogParser-LLM: Advancing Efficient Log Parsing with Large Language Modelscited by 20
  3. 2023RAGLog: Log Anomaly Detection using Retrieval Augmented Generationcited by 19
  4. 2025A Survey of AIOps in the Era of Large Language Modelscited by 18
  5. 2024A Comparative Study on Large Language Models for Log Parsingcited by 5
  6. 2025Mapping the Landscape of Generative AI in Network Monitoring and Managementcited by 4
  7. 2024HELP: Hierarchical Embeddings-based Log Parsingcited by 2
  8. 2024Enhancing Reasoning Capacity of SLM using Cognitive Enhancementcited by 1
  9. 2024Disrupting Test Development with AI Assistantscited by 0
  10. 2025REFLEX: Reference-Free Evaluation of Log Summarization via Large Language Model Judgmentcited by 0
  11. 2026LogRouter: Adaptive Two-Level LLM Routing for Log Question Answering in Big Data Systemscited by 0