# LAnoBERT: System Log Anomaly Detection based on BERT Masked Language Model

arXiv: https://arxiv.org/abs/2111.09564 · HTML: https://mlsanity.com/paper/2111.09564/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 18 Nov 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [LogBERT: Log Anomaly Detection via BERT](https://mlsanity.com/paper/2103.04475/) (2021)
- [Log-based Anomaly Detection Without Log Parsing](https://mlsanity.com/paper/2108.01955/) (2021)
- [Self-Attentive Classification-Based Anomaly Detection in Unstructured Logs](https://mlsanity.com/paper/2008.09340/) (2020)
- [Recurrent Neural Network Attention Mechanisms for Interpretable System Log Anomaly Detection](https://mlsanity.com/paper/1803.04967/) (2018)
- [Detection of Early-Stage Enterprise Infection by Mining Large-Scale Log Data](https://mlsanity.com/paper/1411.5005/) (2014)
- [LSTM-Based System-Call Language Modeling and Robust Ensemble Method for Designing Host-Based Intrusion Detection Systems](https://mlsanity.com/paper/1611.01726/) (2016)

## Built on by (most cited first, 30 of 30)

- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [LogLLM: Log-based Anomaly Detection Using Large Language Models](https://mlsanity.com/paper/2411.08561/) (2024)
- [Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Survey](https://mlsanity.com/paper/2408.07583/) (2024)
- [CAN-BERT do it? Controller Area Network Intrusion Detection System based on BERT Language Model](https://mlsanity.com/paper/2210.09439/) (2022)
- [AI for IT Operations (AIOps) on Cloud Platforms: Reviews, Opportunities and Challenges](https://mlsanity.com/paper/2304.04661/) (2023)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [LogLG: Weakly Supervised Log Anomaly Detection via Log-Event Graph Construction](https://mlsanity.com/paper/2208.10833/) (2022)
- [Log Anomaly Detection with Large Language Models via Knowledge-Enriched Fusion](https://mlsanity.com/paper/2512.11997/) (2025)
- [LLMcap: Large Language Model for Unsupervised PCAP Failure Detection](https://mlsanity.com/paper/2407.06085/) (2024)
- [A Comparative Study on Large Language Models for Log Parsing](https://mlsanity.com/paper/2409.02474/) (2024)
- [FastLogAD: Log Anomaly Detection with Mask-Guided Pseudo Anomaly Generation and Discrimination](https://mlsanity.com/paper/2404.08750/) (2024)
- [AdaptiveLog: An Adaptive Log Analysis Framework with the Collaboration of Large and Small Language Model](https://mlsanity.com/paper/2501.11031/) (2025)
- [LibreLog: Accurate and Efficient Unsupervised Log Parsing Using Open-Source Large Language Models](https://mlsanity.com/paper/2408.01585/) (2024)
- [LogAI: A Library for Log Analytics and Intelligence](https://mlsanity.com/paper/2301.13415/) (2023)
- [LUK: Empowering Log Understanding with Expert Knowledge from Large Language Models](https://mlsanity.com/paper/2409.01909/) (2024)
- [Deep Learning-based Intrusion Detection Systems: A Survey](https://mlsanity.com/paper/2504.07839/) (2025)
- [RuleGenie: SIEM Detection Rule Set Optimization](https://mlsanity.com/paper/2505.06701/) (2025)
- [ADALog: Adaptive Unsupervised Anomaly detection in Logs with Self-attention Masked Language Model](https://mlsanity.com/paper/2505.13496/) (2025)
- [SCMM: Calibrating Cross-modal Representations for Text-Based Person Search](https://mlsanity.com/paper/2304.02278/) (2023)
- [Large Language Models powered Malicious Traffic Detection: Architecture, Opportunities and Case Study](https://mlsanity.com/paper/2503.18487/) (2025)
- [A Novel GPT-Based Framework for Anomaly Detection in System Logs](https://mlsanity.com/paper/2510.16044/) (2025)
- [RAPID: Training-free Retrieval-based Log Anomaly Detection with PLM considering Token-level information](https://mlsanity.com/paper/2311.05160/) (2023)
- [Convolutional vs Large Language Models for Software Log Classification in Edge-Deployable Cellular Network Testing](https://mlsanity.com/paper/2407.03759/) (2024)
- [Diagnosing Robotics Systems Issues with Large Language Models](https://mlsanity.com/paper/2410.09084/) (2024)
- [Good Enough to Learn: LLM-based Anomaly Detection in ECU Logs without Reliable Labels](https://mlsanity.com/paper/2507.01077/) (2025)
- [SeBERTis: A Framework for Producing Classifiers of Security-Related Issue Reports](https://mlsanity.com/paper/2512.15003/) (2025)
- [Debugging Tabular Log as Dynamic Graphs](https://mlsanity.com/paper/2512.22903/) (2025)
- [Log anomaly detection via Meta Learning and Prototypical Networks for Cross domain generalization](https://mlsanity.com/paper/2601.14336/) (2026)
- [Beyond Heavy Log Curation: Perplexity-Based APT Detection via Unsupervised, Context-Augmented Language Models](https://mlsanity.com/paper/2607.20832/) (2026)
- [Exploring Block Anomaly Detection In HDFS Log Data Analysis](https://mlsanity.com/paper/2607.29383/) (2026)

## Abstract (arXiv)

The system log generated in a computer system refers to large-scale data that are collected simultaneously and used as the basic data for determining errors, intrusion and abnormal behaviors. The aim of system log anomaly detection is to promptly identify anomalies while minimizing human intervention, which is a critical problem in the industry. Previous studies performed anomaly detection through algorithms after converting various forms of log data into a standardized template using a parser. Particularly, a template corresponding to a specific event should be defined in advance for all the log data using which the information within the log key may get lost. In this study, we propose LAnoBERT, a parser free system log anomaly detection method that uses the BERT model, exhibiting excellent natural language processing performance. The proposed method, LAnoBERT, learns the model through masked language modeling, which is a BERT-based pre-training method, and proceeds with unsupervised learning-based anomaly detection using the masked language modeling loss function per log key during the test process. In addition, we also propose an efficient inference process to establish a practically applicable pipeline to the actual system. Experiments on three well-known log datasets, i.e., HDFS, BGL, and Thunderbird, show that not only did LAnoBERT yield a higher anomaly detection performance compared to unsupervised learning-based benchmark models, but also it resulted in a comparable performance with supervised learning-based benchmark models.