# Robust and Transferable Anomaly Detection in Log Data using Pre-Trained Language Models

arXiv: https://arxiv.org/abs/2102.11570 · HTML: https://mlsanity.com/paper/2102.11570/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 23 Feb 2021.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Tools and Benchmarks for Automated Log Parsing](https://mlsanity.com/paper/1811.03509/) (2018)
- [Self-Attentive Classification-Based Anomaly Detection in Unstructured Logs](https://mlsanity.com/paper/2008.09340/) (2020)
- [Self-Supervised Log Parsing](https://mlsanity.com/paper/2003.07905/) (2020)
- [Multi-Source Anomaly Detection in Distributed IT Systems](https://mlsanity.com/paper/2101.04977/) (2021)

## Built on by (most cited first, 12 of 12)

- [Deep Learning for Anomaly Detection in Log Data: A Survey](https://mlsanity.com/paper/2207.03820/) (2022)
- [A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directions](https://mlsanity.com/paper/2412.19823/) (2024)
- [A Survey of AIOps in the Era of Large Language Models](https://mlsanity.com/paper/2507.12472/) (2025)
- [AI for IT Operations (AIOps) on Cloud Platforms: Reviews, Opportunities and Challenges](https://mlsanity.com/paper/2304.04661/) (2023)
- [Demystifying and Extracting Fault-indicating Information from Logs for Failure Diagnosis](https://mlsanity.com/paper/2409.13561/) (2024)
- [Mapping the Landscape of Generative AI in Network Monitoring and Management](https://mlsanity.com/paper/2502.08576/) (2025)
- [Inducing Personality in LLM-Based Honeypot Agents: Measuring the Effect on Human-Like Agenda Generation](https://mlsanity.com/paper/2503.19752/) (2025)
- [Enhancing Trust in Autonomous Agents: An Architecture for Accountability and Explainability through Blockchain and Large Language Models](https://mlsanity.com/paper/2403.09567/) (2024)
- [Try with Simpler -- An Evaluation of Improved Principal Component Analysis in Log-based Anomaly Detection](https://mlsanity.com/paper/2308.12612/) (2023)
- [SPEAR: Soft Prompt Enhanced Anomaly Recognition for Time Series Data](https://mlsanity.com/paper/2510.03962/) (2025)
- [Convolutional vs Large Language Models for Software Log Classification in Edge-Deployable Cellular Network Testing](https://mlsanity.com/paper/2407.03759/) (2024)
- [Diagnosing Robotics Systems Issues with Large Language Models](https://mlsanity.com/paper/2410.09084/) (2024)

## Abstract (arXiv)

Anomalies or failures in large computer systems, such as the cloud, have an impact on a large number of users that communicate, compute, and store information. Therefore, timely and accurate anomaly detection is necessary for reliability, security, safe operation, and mitigation of losses in these increasingly important systems. Recently, the evolution of the software industry opens up several problems that need to be tackled including (1) addressing the software evolution due software upgrades, and (2) solving the cold-start problem, where data from the system of interest is not available. In this paper, we propose a framework for anomaly detection in log data, as a major troubleshooting source of system information. To that end, we utilize pre-trained general-purpose language models to preserve the semantics of log messages and map them into log vector embeddings. The key idea is that these representations for the logs are robust and less invariant to changes in the logs, and therefore, result in a better generalization of the anomaly detection models. We perform several experiments on a cloud dataset evaluating different language models for obtaining numerical log representations such as BERT, GPT-2, and XL. The robustness is evaluated by gradually altering log messages, to simulate a change in semantics. Our results show that the proposed approach achieves high performance and robustness, which opens up possibilities for future research in this direction.