# Unsupervised Brain Anomaly Detection and Segmentation with Transformers

arXiv: https://arxiv.org/abs/2102.11650 · HTML: https://mlsanity.com/paper/2102.11650/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 44 references that are in the map.
- First posted on arXiv 23 Feb 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation](https://mlsanity.com/paper/1603.05959/) (2016)
- [Adversarially Learned Inference](https://mlsanity.com/paper/1606.00704/) (2016)
- [Autoencoders for Unsupervised Anomaly Segmentation in Brain MR Images: A Comparative Study](https://mlsanity.com/paper/2004.03271/) (2020)
- [Deep Autoencoding Models for Unsupervised Anomaly Segmentation in Brain MR Images](https://mlsanity.com/paper/1804.04488/) (2018)
- [Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities and Results of the WMH Segmentation Challenge](https://mlsanity.com/paper/1904.00682/) (2019)
- [Unsupervised Lesion Detection via Image Restoration with a Normative Prior](https://mlsanity.com/paper/2005.00031/) (2020)
- [Scale-Space Autoencoders for Unsupervised Anomaly Segmentation in Brain MRI](https://mlsanity.com/paper/2006.12852/) (2020)

## Built on by (most cited first, 30 of 34)

- [Diffusion Models for Medical Anomaly Detection](https://mlsanity.com/paper/2203.04306/) (2022)
- [Fast Unsupervised Brain Anomaly Detection and Segmentation with Diffusion Models](https://mlsanity.com/paper/2206.03461/) (2022)
- [BMAD: Benchmarks for Medical Anomaly Detection](https://mlsanity.com/paper/2306.11876/) (2023)
- [Detecting Outliers with Poisson Image Interpolation](https://mlsanity.com/paper/2107.02622/) (2021)
- [Confidence-based Out-of-Distribution Detection: A Comparative Study and Analysis](https://mlsanity.com/paper/2107.02568/) (2021)
- [Surface Vision Transformers: Attention-Based Modelling applied to Cortical Analysis](https://mlsanity.com/paper/2203.16414/) (2022)
- [Transformer-based out-of-distribution detection for clinically safe segmentation](https://mlsanity.com/paper/2205.10650/) (2022)
- [Data synthesis and adversarial networks: A review and meta-analysis in cancer imaging](https://mlsanity.com/paper/2107.09543/) (2021)
- [Morphology-preserving Autoregressive 3D Generative Modelling of the Brain](https://mlsanity.com/paper/2209.03177/) (2022)
- [medigan: a Python library of pretrained generative models for medical image synthesis](https://mlsanity.com/paper/2209.14472/) (2022)
- [Pretrained ViTs Yield Versatile Representations For Medical Images](https://mlsanity.com/paper/2303.07034/) (2023)
- [On the Pitfalls of Using the Residual Error as Anomaly Score](https://mlsanity.com/paper/2202.03826/) (2022)
- [Transformer based Models for Unsupervised Anomaly Segmentation in Brain MR Images](https://mlsanity.com/paper/2207.02059/) (2022)
- [Cross Attention Transformers for Multi-modal Unsupervised Whole-Body PET Anomaly Detection](https://mlsanity.com/paper/2304.07147/) (2023)
- [What do we learn? Debunking the Myth of Unsupervised Outlier Detection](https://mlsanity.com/paper/2206.03698/) (2022)
- [Conditional diffusion models for guided anomaly detection in brain images using fluid-driven anomaly randomization](https://mlsanity.com/paper/2506.10233/) (2025)
- [Anomaly Detection in Echocardiograms with Dynamic Variational Trajectory Models](https://mlsanity.com/paper/2206.15316/) (2022)
- [Bias in Unsupervised Anomaly Detection in Brain MRI](https://mlsanity.com/paper/2308.13861/) (2023)
- [CRADL: Contrastive Representations for Unsupervised Anomaly Detection and Localization](https://mlsanity.com/paper/2301.02126/) (2023)
- [Adversarial Denoising Diffusion Model for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2312.04382/) (2023)
- [Denoising Diffusion Models for Anomaly Localization in Medical Images](https://mlsanity.com/paper/2410.23834/) (2024)
- [SCOPED: Score-Curvature Out-of-distribution Proximity Evaluator for Diffusion](https://mlsanity.com/paper/2510.01456/) (2025)
- [Transformer-based normative modelling for anomaly detection of early schizophrenia](https://mlsanity.com/paper/2212.04984/) (2022)
- [Enhancing Multi-Class Anomaly Detection via Diffusion Refinement with Dual Conditioning](https://mlsanity.com/paper/2407.01905/) (2024)
- [AutoSeg -- Steering the Inductive Biases for Automatic Pathology Segmentation](https://mlsanity.com/paper/2201.09579/) (2022)
- [Distributional Gaussian Processes Layers for Out-of-Distribution Detection](https://mlsanity.com/paper/2206.13346/) (2022)
- [Unsupervised Anomaly Detection with Local-Sensitive VQVAE and Global-Sensitive Transformers](https://mlsanity.com/paper/2303.17505/) (2023)
- [DisCoPatch: Taming Adversarially-driven Batch Statistics for Improved Out-of-Distribution Detection](https://mlsanity.com/paper/2501.08005/) (2025)
- [Scaling Pretrained Representations Enables Label-Free Out-of-Distribution Detection Without Fine-Tuning](https://mlsanity.com/paper/2605.05638/) (2026)
- [A deep learning approach for detection and localization of leaf anomalies](https://mlsanity.com/paper/2210.03558/) (2022)

## Abstract (arXiv)

Pathological brain appearances may be so heterogeneous as to be intelligible only as anomalies, defined by their deviation from normality rather than any specific pathological characteristic. Amongst the hardest tasks in medical imaging, detecting such anomalies requires models of the normal brain that combine compactness with the expressivity of the complex, long-range interactions that characterise its structural organisation. These are requirements transformers have arguably greater potential to satisfy than other current candidate architectures, but their application has been inhibited by their demands on data and computational resource. Here we combine the latent representation of vector quantised variational autoencoders with an ensemble of autoregressive transformers to enable unsupervised anomaly detection and segmentation defined by deviation from healthy brain imaging data, achievable at low computational cost, within relative modest data regimes. We compare our method to current state-of-the-art approaches across a series of experiments involving synthetic and real pathological lesions. On real lesions, we train our models on 15,000 radiologically normal participants from UK Biobank, and evaluate performance on four different brain MR datasets with small vessel disease, demyelinating lesions, and tumours. We demonstrate superior anomaly detection performance both image-wise and pixel-wise, achievable without post-processing. These results draw attention to the potential of transformers in this most challenging of imaging tasks.