# Unsupervised Anomaly Detection in Medical Images with a Memory-augmented Multi-level Cross-attentional Masked Autoencoder

arXiv: https://arxiv.org/abs/2203.11725 · HTML: https://mlsanity.com/paper/2203.11725/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [PaDiM: a Patch Distribution Modeling Framework for Anomaly Detection and Localization](https://mlsanity.com/paper/2011.08785/) (2020)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [CutPaste: Self-Supervised Learning for Anomaly Detection and Localization](https://mlsanity.com/paper/2104.04015/) (2021)
- [PraNet: Parallel Reverse Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2006.11392/) (2020)
- [COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images](https://mlsanity.com/paper/2003.09871/) (2020)
- [OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representations](https://mlsanity.com/paper/1903.08550/) (2019)
- [PANDA: Adapting Pretrained Features for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2010.05903/) (2020)
- [Deep Anomaly Detection with Deviation Networks](https://mlsanity.com/paper/1911.08623/) (2019)
- [Learning and Evaluating Representations for Deep One-class Classification](https://mlsanity.com/paper/2011.02578/) (2020)
- [Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes](https://mlsanity.com/paper/2111.12264/) (2021)
- [Deep One-Class Classification via Interpolated Gaussian Descriptor](https://mlsanity.com/paper/2101.10043/) (2021)
- [Constrained Contrastive Distribution Learning for Unsupervised Anomaly Detection and Localisation in Medical Images](https://mlsanity.com/paper/2103.03423/) (2021)
- [Exploiting Epistemic Uncertainty of Anatomy Segmentation for Anomaly Detection in Retinal OCT](https://mlsanity.com/paper/1905.12806/) (2019)
- [ACPL: Anti-curriculum Pseudo-labelling for Semi-supervised Medical Image Classification](https://mlsanity.com/paper/2111.12918/) (2021)
- [Scale-Space Autoencoders for Unsupervised Anomaly Segmentation in Brain MRI](https://mlsanity.com/paper/2006.12852/) (2020)
- [Few-Shot Anomaly Detection for Polyp Frames from Colonoscopy](https://mlsanity.com/paper/2006.14811/) (2020)
- [Unsupervised Dual Adversarial Learning for Anomaly Detection in Colonoscopy Video Frames](https://mlsanity.com/paper/1910.10345/) (2019)
- [Self-supervised Mean Teacher for Semi-supervised Chest X-ray Classification](https://mlsanity.com/paper/2103.03629/) (2021)
- [BoMD: Bag of Multi-label Descriptors for Noisy Chest X-ray Classification](https://mlsanity.com/paper/2203.01937/) (2022)
- [∞-former: Infinite Memory Transformer](https://mlsanity.com/paper/2109.00301/) (2021)

## Built on by (most cited first, 15 of 15)

- [Delving into Masked Autoencoders for Multi-Label Thorax Disease Classification](https://mlsanity.com/paper/2210.12843/) (2022)
- [Medical SAM3: A Foundation Model for Universal Prompt-Driven Medical Image Segmentation](https://mlsanity.com/paper/2601.10880/) (2026)
- [FairDomain: Achieving Fairness in Cross-Domain Medical Image Segmentation and Classification](https://mlsanity.com/paper/2407.08813/) (2024)
- [Exploiting Structural Consistency of Chest Anatomy for Unsupervised Anomaly Detection in Radiography Images](https://mlsanity.com/paper/2403.08689/) (2024)
- [Foundation Models and Transformers for Anomaly Detection: A Survey](https://mlsanity.com/paper/2507.15905/) (2025)
- [FairSeg: A Large-Scale Medical Image Segmentation Dataset for Fairness Learning Using Segment Anything Model with Fair Error-Bound Scaling](https://mlsanity.com/paper/2311.02189/) (2023)
- [EDMAE: An Efficient Decoupled Masked Autoencoder for Standard View Identification in Pediatric Echocardiography](https://mlsanity.com/paper/2302.13869/) (2023)
- [ISSTAD: Incremental Self-Supervised Learning Based on Transformer for Anomaly Detection and Localization](https://mlsanity.com/paper/2303.17354/) (2023)
- [Rethinking Polyp Segmentation from an Out-of-Distribution Perspective](https://mlsanity.com/paper/2306.07792/) (2023)
- [Enhancing Reconstruction-Based Out-of-Distribution Detection in Brain MRI with Model and Metric Ensembles](https://mlsanity.com/paper/2412.17586/) (2024)
- [TURBO: The Swiss Knife of Auto-Encoders](https://mlsanity.com/paper/2311.06527/) (2023)
- [Spatial-Aware Self-Supervision for Medical 3D Imaging with Multi-Granularity Observable Tasks](https://mlsanity.com/paper/2509.05967/) (2025)
- [Unsupervised Hybrid framework for ANomaly Detection (HAND) -- applied to Screening Mammogram](https://mlsanity.com/paper/2409.11534/) (2024)
- [Masked Autoencoders for Ultrasound Signals: Robust Representation Learning for Downstream Applications](https://mlsanity.com/paper/2508.20622/) (2025)
- [Momentum-Anchored Multi-Scale Fusion Model for Long-Tailed Chest X-Ray Classification](https://mlsanity.com/paper/2605.02292/) (2026)

## Abstract (arXiv)

Unsupervised anomaly detection (UAD) aims to find anomalous images by optimising a detector using a training set that contains only normal images. UAD approaches can be based on reconstruction methods, self-supervised approaches, and Imagenet pre-trained models. Reconstruction methods, which detect anomalies from image reconstruction errors, are advantageous because they do not rely on the design of problem-specific pretext tasks needed by self-supervised approaches, and on the unreliable translation of models pre-trained from non-medical datasets. However, reconstruction methods may fail because they can have low reconstruction errors even for anomalous images. In this paper, we introduce a new reconstruction-based UAD approach that addresses this low-reconstruction error issue for anomalous images. Our UAD approach, the memory-augmented multi-level cross-attentional masked autoencoder (MemMC-MAE), is a transformer-based approach, consisting of a novel memory-augmented self-attention operator for the encoder and a new multi-level cross-attention operator for the decoder. MemMCMAE masks large parts of the input image during its reconstruction, reducing the risk that it will produce low reconstruction errors because anomalies are likely to be masked and cannot be reconstructed. However, when the anomaly is not masked, then the normal patterns stored in the encoder's memory combined with the decoder's multi-level cross attention will constrain the accurate reconstruction of the anomaly. We show that our method achieves SOTA anomaly detection and localisation on colonoscopy, pneumonia, and covid-19 chest x-ray datasets.