# OODformer: Out-Of-Distribution Detection Transformer

arXiv: https://arxiv.org/abs/2107.08976 · HTML: https://mlsanity.com/paper/2107.08976/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 56 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances](https://mlsanity.com/paper/2007.08176/) (2020)
- [Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data](https://mlsanity.com/paper/2002.11297/) (2020)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [Large-Margin Softmax Loss for Convolutional Neural Networks](https://mlsanity.com/paper/1612.02295/) (2016)
- [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)

## Built on by (most cited first, 27 of 27)

- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges](https://mlsanity.com/paper/2110.14051/) (2021)
- [In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation](https://mlsanity.com/paper/2306.00826/) (2023)
- [CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No](https://mlsanity.com/paper/2308.12213/) (2023)
- [LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning](https://mlsanity.com/paper/2306.01293/) (2023)
- [Relationformer: A Unified Framework for Image-to-Graph Generation](https://mlsanity.com/paper/2203.10202/) (2022)
- [Nearest Neighbor Guidance for Out-of-Distribution Detection](https://mlsanity.com/paper/2309.14888/) (2023)
- [Out-of-distribution Detection in Medical Image Analysis: A survey](https://mlsanity.com/paper/2404.18279/) (2024)
- [Reconstruction-based Out-of-Distribution Detection for Short-Range FMCW Radar](https://mlsanity.com/paper/2302.14192/) (2023)
- [Data Invariants to Understand Unsupervised Out-of-Distribution Detection](https://mlsanity.com/paper/2111.13362/) (2021)
- [Is it all a cluster game? -- Exploring Out-of-Distribution Detection based on Clustering in the Embedding Space](https://mlsanity.com/paper/2203.08549/) (2022)
- [Box Supervised Video Segmentation Proposal Network](https://mlsanity.com/paper/2202.07025/) (2022)
- [A Baseline for Detecting Out-of-Distribution Examples in Image Captioning](https://mlsanity.com/paper/2207.05418/) (2022)
- [HOOD: Real-Time Human Presence and Out-of-Distribution Detection Using FMCW Radar](https://mlsanity.com/paper/2308.02396/) (2023)
- [How to Handle Different Types of Out-of-Distribution Scenarios in Computational Argumentation? A Comprehensive and Fine-Grained Field Study](https://mlsanity.com/paper/2309.08316/) (2023)
- [Can Pre-trained Networks Detect Familiar Out-of-Distribution Data?](https://mlsanity.com/paper/2310.00847/) (2023)
- [Fast and Interpretable Face Identification for Out-Of-Distribution Data Using Vision Transformers](https://mlsanity.com/paper/2311.02803/) (2023)
- [Self-Supervised Anomaly Detection by Self-Distillation and Negative Sampling](https://mlsanity.com/paper/2201.06378/) (2022)
- [No Shifted Augmentations (NSA): compact distributions for robust self-supervised Anomaly Detection](https://mlsanity.com/paper/2203.10344/) (2022)
- [PViT: Prior-augmented Vision Transformer for Out-of-distribution Detection](https://mlsanity.com/paper/2410.20631/) (2024)
- [Open Set Recognition using Vision Transformer with an Additional Detection Head](https://mlsanity.com/paper/2203.08441/) (2022)
- [Semantic Novelty Detection via Relational Reasoning](https://mlsanity.com/paper/2207.08699/) (2022)
- [How Out-of-Distribution Detection Learning Theory Enhances Transformer: Learnability and Reliability](https://mlsanity.com/paper/2406.12915/) (2024)
- [A Latent Space Correlation-Aware Autoencoder for Anomaly Detection in Skewed Data](https://mlsanity.com/paper/2301.00462/) (2023)
- [Can Dense Connectivity Benefit Outlier Detection? An Odyssey with NAS](https://mlsanity.com/paper/2406.01975/) (2024)
- [CODE: Cross-Modal Calibration and Dynamic Suppression for Open World Object Detection](https://mlsanity.com/paper/2608.27214/) (2026)

## Abstract (arXiv)

A serious problem in image classification is that a trained model might perform well for input data that originates from the same distribution as the data available for model training, but performs much worse for out-of-distribution (OOD) samples. In real-world safety-critical applications, in particular, it is important to be aware if a new data point is OOD. To date, OOD detection is typically addressed using either confidence scores, auto-encoder based reconstruction, or by contrastive learning. However, the global image context has not yet been explored to discriminate the non-local objectness between in-distribution and OOD samples. This paper proposes a first-of-its-kind OOD detection architecture named OODformer that leverages the contextualization capabilities of the transformer. Incorporating the trans-former as the principal feature extractor allows us to exploit the object concepts and their discriminate attributes along with their co-occurrence via visual attention. Using the contextualised embedding, we demonstrate OOD detection using both class-conditioned latent space similarity and a network confidence score. Our approach shows improved generalizability across various datasets. We have achieved a new state-of-the-art result on CIFAR-10/-100 and ImageNet30.