# Benchmarking Large Language Models for Log Analysis, Security, and Interpretation

arXiv: https://arxiv.org/abs/2311.14519 · HTML: https://mlsanity.com/paper/2311.14519/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2023.

## Built on by (most cited first, 19 of 19)

- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [Audit-LLM: Multi-Agent Collaboration for Log-based Insider Threat Detection](https://mlsanity.com/paper/2408.08902/) (2024)
- [LogEval: A Comprehensive Benchmark Suite for Large Language Models In Log Analysis](https://mlsanity.com/paper/2407.01896/) (2024)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiency](https://mlsanity.com/paper/2402.19366/) (2024)
- [ProveRAG: Provenance-Driven Vulnerability Analysis with Automated Retrieval-Augmented LLMs](https://mlsanity.com/paper/2410.17406/) (2024)
- [Evaluating Language Models For Threat Detection in IoT Security Logs](https://mlsanity.com/paper/2507.02390/) (2025)
- [DiagChain: A Diagnostic Benchmark for Evaluating LLM Agents on Evidence-Grounded Attack Chain Reconstruction](https://mlsanity.com/paper/2608.03591/) (2026)
- [Evaluating Large Language Models in Vulnerability Detection Under Variable Context Windows](https://mlsanity.com/paper/2502.00064/) (2025)
- [LLM Embedding-based Attribution (LEA): Quantifying Source Contributions to Generative Model's Response for Vulnerability Analysis](https://mlsanity.com/paper/2506.12100/) (2025)
- [CAM-LDS: Cyber Attack Manifestations for Automatic Interpretation of System Logs and Security Alerts](https://mlsanity.com/paper/2603.04186/) (2026)
- [LogBabylon: A Unified Framework for Cross-Log File Integration and Analysis](https://mlsanity.com/paper/2412.12364/) (2024)
- [DMFI: A Dual-Modality Log Analysis Framework for Insider Threat Detection with LoRA-Tuned Language Models](https://mlsanity.com/paper/2508.05694/) (2025)
- [Towards Agentic Investigation of Security Alerts](https://mlsanity.com/paper/2604.25846/) (2026)
- [Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques](https://mlsanity.com/paper/2507.13629/) (2025)
- [A global log for medical AI](https://mlsanity.com/paper/2510.04033/) (2025)
- [A Hybrid Insider Threat Detection Framework Combining Multi-Agent Simulation, Layered SIEM Correlation, and Theory-of-Mind Reasoning](https://mlsanity.com/paper/2601.04243/) (2026)
- [Fine-Tuning Small Language Models for Solution-Oriented Windows Event Log Analysis](https://mlsanity.com/paper/2605.06330/) (2026)

## Abstract (arXiv)

Large Language Models (LLM) continue to demonstrate their utility in a variety of emergent capabilities in different fields. An area that could benefit from effective language understanding in cybersecurity is the analysis of log files. This work explores LLMs with different architectures (BERT, RoBERTa, DistilRoBERTa, GPT-2, and GPT-Neo) that are benchmarked for their capacity to better analyze application and system log files for security. Specifically, 60 fine-tuned language models for log analysis are deployed and benchmarked. The resulting models demonstrate that they can be used to perform log analysis effectively with fine-tuning being particularly important for appropriate domain adaptation to specific log types. The best-performing fine-tuned sequence classification model (DistilRoBERTa) outperforms the current state-of-the-art; with an average F1-Score of 0.998 across six datasets from both web application and system log sources. To achieve this, we propose and implement a new experimentation pipeline (LLM4Sec) which leverages LLMs for log analysis experimentation, evaluation, and analysis.