# ISSTAD: Incremental Self-Supervised Learning Based on Transformer for Anomaly Detection and Localization

arXiv: https://arxiv.org/abs/2303.17354 · HTML: https://mlsanity.com/paper/2303.17354/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 62 references that are in the map.
- First posted on arXiv 30 Mar 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [CutPaste: Self-Supervised Learning for Anomaly Detection and Localization](https://mlsanity.com/paper/2104.04015/) (2021)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [GANomaly: Semi-Supervised Anomaly Detection via Adversarial Training](https://mlsanity.com/paper/1805.06725/) (2018)
- [SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2207.14315/) (2022)
- [Deep Learning for Anomaly Detection: A Survey](https://mlsanity.com/paper/1901.03407/) (2019)
- [Context Autoencoder for Self-Supervised Representation Learning](https://mlsanity.com/paper/2202.03026/) (2022)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [Patch SVDD: Patch-level SVDD for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2006.16067/) (2020)
- [OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representations](https://mlsanity.com/paper/1903.08550/) (2019)
- [A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks](https://mlsanity.com/paper/2011.01844/) (2020)
- [Skip-GANomaly: Skip Connected and Adversarially Trained Encoder-Decoder Anomaly Detection](https://mlsanity.com/paper/1901.08954/) (2019)
- [A Survey of Deep Meta-Learning](https://mlsanity.com/paper/2010.03522/) (2020)
- [Omni-frequency Channel-selection Representations for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2203.00259/) (2022)
- [Anomaly Detection in Medical Imaging -- A Mini Review](https://mlsanity.com/paper/2108.11986/) (2021)
- [Self-Supervised Masking for Unsupervised Anomaly Detection and Localization](https://mlsanity.com/paper/2205.06568/) (2022)
- [AnoViT: Unsupervised Anomaly Detection and Localization with Vision Transformer-based Encoder-Decoder](https://mlsanity.com/paper/2203.10808/) (2022)
- [Towards Automatic Threat Detection: A Survey of Advances of Deep Learning within X-ray Security Imaging](https://mlsanity.com/paper/2001.01293/) (2020)
- [Visual Anomaly Detection Via Partition Memory Bank Module and Error Estimation](https://mlsanity.com/paper/2209.12441/) (2022)
- [Unsupervised Anomaly Detection in Medical Images with a Memory-augmented Multi-level Cross-attentional Masked Autoencoder](https://mlsanity.com/paper/2203.11725/) (2022)
- [Deep CNN-based Multi-task Learning for Open-Set Recognition](https://mlsanity.com/paper/1903.03161/) (2019)
- [Deep Variational Semi-Supervised Novelty Detection](https://mlsanity.com/paper/1911.04971/) (2019)
- [Transformation Based Deep Anomaly Detection in Astronomical Images](https://mlsanity.com/paper/2005.07779/) (2020)

## Built on by (most cited first, 4 of 4)

- [BMAD: Benchmarks for Medical Anomaly Detection](https://mlsanity.com/paper/2306.11876/) (2023)
- [Few-Shot Learning in Video and 3D Object Detection: A Survey](https://mlsanity.com/paper/2507.17079/) (2025)
- [Decoupled Complementary Spectral-Spatial Learning for Background Representation Enhancement in Hyperspectral Anomaly Detection](https://mlsanity.com/paper/2510.18781/) (2025)
- [Semantic-Deviation-Anchored Multi-Branch Fusion for Unsupervised Anomaly Detection and Localization in Unstructured Conveyor-Belt Coal Scenes](https://mlsanity.com/paper/2602.07694/) (2026)

## Abstract (arXiv)

In the realm of machine learning, the study of anomaly detection and localization within image data has gained substantial traction, particularly for practical applications such as industrial defect detection. While the majority of existing methods predominantly use Convolutional Neural Networks (CNN) as their primary network architecture, we introduce a novel approach based on the Transformer backbone network. Our method employs a two-stage incremental learning strategy. During the first stage, we train a Masked Autoencoder (MAE) model solely on normal images. In the subsequent stage, we apply pixel-level data augmentation techniques to generate corrupted normal images and their corresponding pixel labels. This process allows the model to learn how to repair corrupted regions and classify the status of each pixel. Ultimately, the model generates a pixel reconstruction error matrix and a pixel anomaly probability matrix. These matrices are then combined to produce an anomaly scoring matrix that effectively detects abnormal regions. When benchmarked against several state-of-the-art CNN-based methods, our approach exhibits superior performance on the MVTec AD dataset, achieving an impressive 97.6% AUC.