# Self-Supervised Masking for Unsupervised Anomaly Detection and Localization

arXiv: https://arxiv.org/abs/2205.06568 · HTML: https://mlsanity.com/paper/2205.06568/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 84 references that are in the map.
- First posted on arXiv 13 May 2022.

## Builds on (most cited first, 30 of 37)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [CutPaste: Self-Supervised Learning for Anomaly Detection and Localization](https://mlsanity.com/paper/2104.04015/) (2021)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddings](https://mlsanity.com/paper/1911.02357/) (2019)
- [Multiresolution Knowledge Distillation for Anomaly Detection](https://mlsanity.com/paper/2011.11108/) (2020)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [Adversarially Learned One-Class Classifier for Novelty Detection](https://mlsanity.com/paper/1802.09088/) (2018)
- [OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representations](https://mlsanity.com/paper/1903.08550/) (2019)
- [Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index](https://mlsanity.com/paper/1308.3052/) (2013)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [Deep Autoencoding Models for Unsupervised Anomaly Segmentation in Brain MR Images](https://mlsanity.com/paper/1804.04488/) (2018)
- [Efficient GAN-Based Anomaly Detection](https://mlsanity.com/paper/1802.06222/) (2018)
- [Skip-GANomaly: Skip Connected and Adversarially Trained Encoder-Decoder Anomaly Detection](https://mlsanity.com/paper/1901.08954/) (2019)
- [Mask Guided Matting via Progressive Refinement Network](https://mlsanity.com/paper/2012.06722/) (2020)
- [Encoding Structure-Texture Relation with P-Net for Anomaly Detection in Retinal Images](https://mlsanity.com/paper/2008.03632/) (2020)
- [Deep-Anomaly: Fully Convolutional Neural Network for Fast Anomaly Detection in Crowded Scenes](https://mlsanity.com/paper/1609.00866/) (2016)
- [Iterative energy-based projection on a normal data manifold for anomaly localization](https://mlsanity.com/paper/2002.03734/) (2020)
- [An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos](https://mlsanity.com/paper/1801.03149/) (2018)
- [Self-Supervised Feature Learning by Learning to Spot Artifacts](https://mlsanity.com/paper/1806.05024/) (2018)
- [Anomaly Detection in Medical Imaging with Deep Perceptual Autoencoders](https://mlsanity.com/paper/2006.13265/) (2020)

## Built on by (most cited first, 30 of 43)

- [Deep Industrial Image Anomaly Detection: A Survey](https://mlsanity.com/paper/2301.11514/) (2023)
- [Registration based Few-Shot Anomaly Detection](https://mlsanity.com/paper/2207.07361/) (2022)
- [Segment Any Anomaly without Training via Hybrid Prompt Regularization](https://mlsanity.com/paper/2305.10724/) (2023)
- [Label-Free Liver Tumor Segmentation](https://mlsanity.com/paper/2303.14869/) (2023)
- [Self-Distilled Masked Auto-Encoders are Efficient Video Anomaly Detectors](https://mlsanity.com/paper/2306.12041/) (2023)
- [PO3AD: Predicting Point Offsets toward Better 3D Point Cloud Anomaly Detection](https://mlsanity.com/paper/2412.12617/) (2024)
- [Anomaly Heterogeneity Learning for Open-set Supervised Anomaly Detection](https://mlsanity.com/paper/2310.12790/) (2023)
- [AMI-Net: Adaptive Mask Inpainting Network for Industrial Anomaly Detection and Localization](https://mlsanity.com/paper/2412.11802/) (2024)
- [Combating Representation Learning Disparity with Geometric Harmonization](https://mlsanity.com/paper/2310.17622/) (2023)
- [Recurrent Generic Contour-based Instance Segmentation with Progressive Learning](https://mlsanity.com/paper/2301.08898/) (2023)
- [Learning Multi-view Anomaly Detection with Efficient Adaptive Selection](https://mlsanity.com/paper/2407.11935/) (2024)
- [Few-Shot Anomaly Detection via Category-Agnostic Registration Learning](https://mlsanity.com/paper/2406.08810/) (2024)
- [Variational Autoencoder for Anomaly Detection: A Comparative Study](https://mlsanity.com/paper/2408.13561/) (2024)
- [VQ-Flow: Taming Normalizing Flows for Multi-Class Anomaly Detection via Hierarchical Vector Quantization](https://mlsanity.com/paper/2409.00942/) (2024)
- [ISSTAD: Incremental Self-Supervised Learning Based on Transformer for Anomaly Detection and Localization](https://mlsanity.com/paper/2303.17354/) (2023)
- [Self-Supervised Training with Autoencoders for Visual Anomaly Detection](https://mlsanity.com/paper/2206.11723/) (2022)
- [Multi-Scale Memory Comparison for Zero-/Few-Shot Anomaly Detection](https://mlsanity.com/paper/2308.04789/) (2023)
- [IPAD: Industrial Process Anomaly Detection Dataset](https://mlsanity.com/paper/2404.15033/) (2024)
- [A Semantically Disentangled Unified Model for Multi-category 3D Anomaly Detection](https://mlsanity.com/paper/2603.25159/) (2026)
- [Improved Anomaly Detection by Using the Attention-Based Isolation Forest](https://mlsanity.com/paper/2210.02558/) (2022)
- [SaliencyCut: Augmenting Plausible Anomalies for Anomaly Detection](https://mlsanity.com/paper/2306.08366/) (2023)
- [Autoencoder-Based Visual Anomaly Localization for Manufacturing Quality Control](https://mlsanity.com/paper/2309.06884/) (2023)
- [Excision And Recovery: Visual Defect Obfuscation Based Self-Supervised Anomaly Detection Strategy](https://mlsanity.com/paper/2310.04010/) (2023)
- [OpenSlot: Mixed Open-Set Recognition with Object-Centric Learning](https://mlsanity.com/paper/2407.02386/) (2024)
- [Self-Supervised Learning for Real-World Object Detection: a Survey](https://mlsanity.com/paper/2410.07442/) (2024)
- [A Comprehensive Survey for Real-World Industrial Defect Detection: Challenges, Approaches, and Prospects](https://mlsanity.com/paper/2507.13378/) (2025)
- [Q-Former Autoencoder: A Modern Framework for Medical Anomaly Detection](https://mlsanity.com/paper/2507.18481/) (2025)
- [Chili Pepper Disease Diagnosis via Image Reconstruction Using GrabCut and Generative Adversarial Serial Autoencoder](https://mlsanity.com/paper/2306.12057/) (2023)
- [Weakly Supervised Anomaly Detection for Chest X-Ray Image](https://mlsanity.com/paper/2311.09642/) (2023)
- [Multilevel Saliency-Guided Self-Supervised Learning for Image Anomaly Detection](https://mlsanity.com/paper/2311.18332/) (2023)

## Abstract (arXiv)

Recently, anomaly detection and localization in multimedia data have received significant attention among the machine learning community. In real-world applications such as medical diagnosis and industrial defect detection, anomalies only present in a fraction of the images. To extend the reconstruction-based anomaly detection architecture to the localized anomalies, we propose a self-supervised learning approach through random masking and then restoring, named Self-Supervised Masking (SSM) for unsupervised anomaly detection and localization. SSM not only enhances the training of the inpainting network but also leads to great improvement in the efficiency of mask prediction at inference. Through random masking, each image is augmented into a diverse set of training triplets, thus enabling the autoencoder to learn to reconstruct with masks of various sizes and shapes during training. To improve the efficiency and effectiveness of anomaly detection and localization at inference, we propose a novel progressive mask refinement approach that progressively uncovers the normal regions and finally locates the anomalous regions. The proposed SSM method outperforms several state-of-the-arts for both anomaly detection and anomaly localization, achieving 98.3% AUC on Retinal-OCT and 93.9% AUC on MVTec AD, respectively.