# Transformer-based out-of-distribution detection for clinically safe segmentation

arXiv: https://arxiv.org/abs/2205.10650 · HTML: https://mlsanity.com/paper/2205.10650/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 21 May 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [A large annotated medical image dataset for the development and evaluation of segmentation algorithms](https://mlsanity.com/paper/1902.09063/) (2019)
- [Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation](https://mlsanity.com/paper/1911.13273/) (2019)
- [Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation](https://mlsanity.com/paper/1808.01200/) (2018)
- [Unsupervised Brain Anomaly Detection and Segmentation with Transformers](https://mlsanity.com/paper/2102.11650/) (2021)
- [Knowing what you know in brain segmentation using Bayesian deep neural networks](https://mlsanity.com/paper/1812.01719/) (2018)
- [Uncertainty-driven Sanity Check: Application to Postoperative Brain Tumor Cavity Segmentation](https://mlsanity.com/paper/1806.03106/) (2018)
- [Neuromorphologicaly-preserving Volumetric data encoding using VQ-VAE](https://mlsanity.com/paper/2002.05692/) (2020)

## Built on by (most cited first, 16 of 16)

- [Fast Unsupervised Brain Anomaly Detection and Segmentation with Diffusion Models](https://mlsanity.com/paper/2206.03461/) (2022)
- [Generative AI for Medical Imaging: extending the MONAI Framework](https://mlsanity.com/paper/2307.15208/) (2023)
- [Unsupervised 3D out-of-distribution detection with latent diffusion models](https://mlsanity.com/paper/2307.03777/) (2023)
- [Out-of-distribution Detection in Medical Image Analysis: A survey](https://mlsanity.com/paper/2404.18279/) (2024)
- [Morphology-preserving Autoregressive 3D Generative Modelling of the Brain](https://mlsanity.com/paper/2209.03177/) (2022)
- [Comparative Benchmarking of Failure Detection Methods in Medical Image Segmentation: Unveiling the Role of Confidence Aggregation](https://mlsanity.com/paper/2406.03323/) (2024)
- [Dissecting Sample Hardness: A Fine-Grained Analysis of Hardness Characterization Methods for Data-Centric AI](https://mlsanity.com/paper/2403.04551/) (2024)
- [SynergyNet: Bridging the Gap between Discrete and Continuous Representations for Precise Medical Image Segmentation](https://mlsanity.com/paper/2310.17764/) (2023)
- [Enhancing Reconstruction-Based Out-of-Distribution Detection in Brain MRI with Model and Metric Ensembles](https://mlsanity.com/paper/2412.17586/) (2024)
- [MedLoRD: A Medical Low-Resource Diffusion Model for High-Resolution 3D CT Image Synthesis](https://mlsanity.com/paper/2503.13211/) (2025)
- [Transformer-based normative modelling for anomaly detection of early schizophrenia](https://mlsanity.com/paper/2212.04984/) (2022)
- [Quantifying uncertainty in lung cancer segmentation with foundation models applied to mixed-domain datasets](https://mlsanity.com/paper/2403.13113/) (2024)
- [How Out-of-Distribution Detection Learning Theory Enhances Transformer: Learnability and Reliability](https://mlsanity.com/paper/2406.12915/) (2024)
- [Random forest-based out-of-distribution detection for robust lung cancer segmentation](https://mlsanity.com/paper/2508.19112/) (2025)
- [Out-of-Distribution Radar Detection with Complex VAEs: Theory, Whitening, and ANMF Fusion](https://mlsanity.com/paper/2601.18677/) (2026)
- [Unsupervised Anomaly Detection for Image Dataset Quality Assurance in Multi-Center Breast MRI](https://mlsanity.com/paper/2608.16725/) (2026)

## Abstract (arXiv)

In a clinical setting it is essential that deployed image processing systems are robust to the full range of inputs they might encounter and, in particular, do not make confidently wrong predictions. The most popular approach to safe processing is to train networks that can provide a measure of their uncertainty, but these tend to fail for inputs that are far outside the training data distribution. Recently, generative modelling approaches have been proposed as an alternative; these can quantify the likelihood of a data sample explicitly, filtering out any out-of-distribution (OOD) samples before further processing is performed. In this work, we focus on image segmentation and evaluate several approaches to network uncertainty in the far-OOD and near-OOD cases for the task of segmenting haemorrhages in head CTs. We find all of these approaches are unsuitable for safe segmentation as they provide confidently wrong predictions when operating OOD. We propose performing full 3D OOD detection using a VQ-GAN to provide a compressed latent representation of the image and a transformer to estimate the data likelihood. Our approach successfully identifies images in both the far- and near-OOD cases. We find a strong relationship between image likelihood and the quality of a model's segmentation, making this approach viable for filtering images unsuitable for segmentation. To our knowledge, this is the first time transformers have been applied to perform OOD detection on 3D image data. Code is available at github.com/marksgraham/transformer-ood.