# AnomalyBERT: Self-Supervised Transformer for Time Series Anomaly Detection using Data Degradation Scheme

arXiv: https://arxiv.org/abs/2305.04468 · HTML: https://mlsanity.com/paper/2305.04468/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 8 May 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding](https://mlsanity.com/paper/1802.04431/) (2018)
- [Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy](https://mlsanity.com/paper/2110.02642/) (2021)
- [Graph Neural Network-Based Anomaly Detection in Multivariate Time Series](https://mlsanity.com/paper/2106.06947/) (2021)
- [Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications](https://mlsanity.com/paper/1802.03903/) (2018)
- [A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencoder](https://mlsanity.com/paper/1711.00614/) (2017)
- [A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data](https://mlsanity.com/paper/1811.08055/) (2018)
- [Towards a Rigorous Evaluation of Time-series Anomaly Detection](https://mlsanity.com/paper/2109.05257/) (2021)

## Built on by (most cited first, 25 of 25)

- [Large Language Model Guided Knowledge Distillation for Time Series Anomaly Detection](https://mlsanity.com/paper/2401.15123/) (2024)
- [DACAD: Domain Adaptation Contrastive Learning for Anomaly Detection in Multivariate Time Series](https://mlsanity.com/paper/2404.11269/) (2024)
- [Robust and Explainable Detector of Time Series Anomaly via Augmenting Multiclass Pseudo-Anomalies](https://mlsanity.com/paper/2505.20765/) (2025)
- [Mamba Adaptive Anomaly Transformer with association discrepancy for time series](https://mlsanity.com/paper/2502.07858/) (2025)
- [A Review on Self-Supervised Learning for Time Series Anomaly Detection: Recent Advances and Open Challenges](https://mlsanity.com/paper/2501.15196/) (2025)
- [An Encode-then-Decompose Approach to Unsupervised Time Series Anomaly Detection on Contaminated Training Data--Extended Version](https://mlsanity.com/paper/2510.18998/) (2025)
- [A Comparison of Deep Learning Architectures for Spacecraft Anomaly Detection](https://mlsanity.com/paper/2403.12864/) (2024)
- [A New Perspective on Time Series Anomaly Detection: Faster Patch-based Broad Learning System](https://mlsanity.com/paper/2412.05498/) (2024)
- [TriP-LLM: A Tri-Branch Patch-wise Large Language Model Framework for Time-Series Anomaly Detection](https://mlsanity.com/paper/2508.00047/) (2025)
- [CaPulse: Detecting Anomalies by Tuning in to the Causal Rhythms of Time Series](https://mlsanity.com/paper/2508.04630/) (2025)
- [Attention in Motion: Secure Platooning via Transformer-based Misbehavior Detection](https://mlsanity.com/paper/2512.15503/) (2025)
- [U-shaped Transformer: Retain High Frequency Context in Time Series Analysis](https://mlsanity.com/paper/2307.09019/) (2023)
- [TCGPN: Temporal-Correlation Graph Pre-trained Network for Stock Forecasting](https://mlsanity.com/paper/2407.18519/) (2024)
- [Learning to Hear Broken Motors: Signature-Guided Data Augmentation for Induction-Motor Diagnostics](https://mlsanity.com/paper/2506.08412/) (2025)
- [CAPMix: Robust KPI Anomaly Detection for AIOps in Noisy and Dynamic Environments](https://mlsanity.com/paper/2509.06419/) (2025)
- [Beyond Marginals: Learning Joint Spatio-Temporal Patterns for Multivariate Anomaly Detection](https://mlsanity.com/paper/2509.15033/) (2025)
- [Pi-transformer: A prior-informed dual-attention model for multivariate time-series anomaly detection](https://mlsanity.com/paper/2509.19985/) (2025)
- [Dynamic Reward Scaling for Multivariate Time Series Anomaly Detection: A VAE-Enhanced Reinforcement Learning Approach](https://mlsanity.com/paper/2511.12351/) (2025)
- [SCG With Your Phone: Diagnosis of Rhythmic Spectrum Disorders in Field Conditions](https://mlsanity.com/paper/2601.13926/) (2026)
- [Bridging Classification and Reconstruction: Cooperative Time Series Anomaly Detection](https://mlsanity.com/paper/2605.26193/) (2026)
- [Benchmarking Inductive Biases for Multivariate Time-Series Anomaly Detection with a Robust Multi-View Channel-Graph Detector](https://mlsanity.com/paper/2605.28103/) (2026)
- [Optimizing Transformer Neural Network for Real-Time Outlier Detection on FPGAs](https://mlsanity.com/paper/2607.22786/) (2026)
- [Multi-Modal Anomaly Detection: A Survey](https://mlsanity.com/paper/2608.24937/) (2026)
- [Supervision Recovery for Time Series Anomaly Detection via Context-Anchored Pairing](https://mlsanity.com/paper/2609.33610/) (2026)
- [Generalist Representation, Specialist Detection: TS-Router for Time-Series Anomaly Detection](https://mlsanity.com/paper/2610.00978/) (2026)

## Abstract (arXiv)

Mechanical defects in real situations affect observation values and cause abnormalities in multivariate time series, such as sensor values or network data. To perceive abnormalities in such data, it is crucial to understand the temporal context and interrelation between variables simultaneously. The anomaly detection task for time series, especially for unlabeled data, has been a challenging problem, and we address it by applying a suitable data degradation scheme to self-supervised model training. We define four types of synthetic outliers and propose the degradation scheme in which a portion of input data is replaced with one of the synthetic outliers. Inspired by the self-attention mechanism, we design a Transformer-based architecture to recognize the temporal context and detect unnatural sequences with high efficiency. Our model converts multivariate data points into temporal representations with relative position bias and yields anomaly scores from these representations. Our method, AnomalyBERT, shows a great capability of detecting anomalies contained in complex time series and surpasses previous state-of-the-art methods on five real-world benchmarks. Our code is available at https://github.com/Jhryu30/AnomalyBERT.