# LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection

arXiv: https://arxiv.org/abs/2401.04749 · HTML: https://mlsanity.com/paper/2401.04749/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 26 references that are in the map.
- First posted on arXiv 9 Jan 2024.

## Builds on (most cited first, 5 of 5)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation](https://mlsanity.com/paper/2210.15461/) (2022)

## Built on by (most cited first, 15 of 15)

- [mABC: multi-Agent Blockchain-Inspired Collaboration for root cause analysis in micro-services architecture](https://mlsanity.com/paper/2404.12135/) (2024)
- [E^2-LLM: Efficient and Extreme Length Extension of Large Language Models](https://mlsanity.com/paper/2401.06951/) (2024)
- [Adapting Large Language Models for Parameter-Efficient Log Anomaly Detection](https://mlsanity.com/paper/2503.08045/) (2025)
- [ADALog: Adaptive Unsupervised Anomaly detection in Logs with Self-attention Masked Language Model](https://mlsanity.com/paper/2505.13496/) (2025)
- [LabelCoRank: Revolutionizing Long Tail Multi-Label Classification with Co-Occurrence Reranking](https://mlsanity.com/paper/2503.07968/) (2025)
- [Diagnosing and Resolving Cloud Platform Instability with Multi-modal RAG LLMs](https://mlsanity.com/paper/2505.21419/) (2025)
- [MINES: Explainable Anomaly Detection through Web API Invariant Inference](https://mlsanity.com/paper/2512.06906/) (2025)
- [EventADL: Open-Box Anomaly Detection and Localization Framework for Events in Cloud-Based Service Systems](https://mlsanity.com/paper/2605.00936/) (2026)
- [MLAD: A Unified Model for Multi-system Log Anomaly Detection](https://mlsanity.com/paper/2401.07655/) (2024)
- [What Information Contributes to Log-based Anomaly Detection? Insights from a Configurable Transformer-Based Approach](https://mlsanity.com/paper/2409.20503/) (2024)
- [FusionLog: Cross-System Log-based Anomaly Detection via Fusion of General and Proprietary Knowledge](https://mlsanity.com/paper/2511.05878/) (2025)
- [A unified framework for detecting point and collective anomalies in operating system logs via collaborative transformers](https://mlsanity.com/paper/2512.23380/) (2025)
- [Beyond Capability Benchmarks: Learning Operational Fingerprints of LLM Cloud Services from Production Incident Metadata](https://mlsanity.com/paper/2608.26332/) (2026)
- [Supervision Recovery for Time Series Anomaly Detection via Context-Anchored Pairing](https://mlsanity.com/paper/2609.33610/) (2026)
- [Generalist Representation, Specialist Detection: TS-Router for Time-Series Anomaly Detection](https://mlsanity.com/paper/2610.00978/) (2026)

## Abstract (arXiv)

Log anomaly detection is a key component in the field of artificial intelligence for IT operations (AIOps). Considering log data of variant domains, retraining the whole network for unknown domains is inefficient in real industrial scenarios. However, previous deep models merely focused on extracting the semantics of log sequences in the same domain, leading to poor generalization on multi-domain logs. To alleviate this issue, we propose a unified Transformer-based framework for Log anomaly detection (LogFormer) to improve the generalization ability across different domains, where we establish a two-stage process including the pre-training and adapter-based tuning stage. Specifically, our model is first pre-trained on the source domain to obtain shared semantic knowledge of log data. Then, we transfer such knowledge to the target domain via shared parameters. Besides, the Log-Attention module is proposed to supplement the information ignored by the log-paring. The proposed method is evaluated on three public and one real-world datasets. Experimental results on multiple benchmarks demonstrate the effectiveness of our LogFormer with fewer trainable parameters and lower training costs.