# Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP

arXiv: https://arxiv.org/abs/2109.02748 · HTML: https://mlsanity.com/paper/2109.02748/

## Key facts

- Cited by 152 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 6 Sep 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances](https://mlsanity.com/paper/2007.08176/) (2020)
- [Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data](https://mlsanity.com/paper/2002.11297/) (2020)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [Pretrained Transformers Improve Out-of-Distribution Robustness](https://mlsanity.com/paper/2004.06100/) (2020)
- [Exploring the Limits of Out-of-Distribution Detection](https://mlsanity.com/paper/2106.03004/) (2021)
- [Professor Forcing: A New Algorithm for Training Recurrent Networks](https://mlsanity.com/paper/1610.09038/) (2016)
- [Contrastive Training for Improved Out-of-Distribution Detection](https://mlsanity.com/paper/2007.05566/) (2020)
- [Generative OpenMax for Multi-Class Open Set Classification](https://mlsanity.com/paper/1707.07418/) (2017)
- [C2AE: Class Conditioned Auto-Encoder for Open-set Recognition](https://mlsanity.com/paper/1904.01198/) (2019)
- [DOC: Deep Open Classification of Text Documents](https://mlsanity.com/paper/1709.08716/) (2017)
- [Open-world Learning and Application to Product Classification](https://mlsanity.com/paper/1809.06004/) (2018)
- [Unseen Class Discovery in Open-world Classification](https://mlsanity.com/paper/1801.05609/) (2018)
- [Memory Based Online Learning of Deep Representations from Video Streams](https://mlsanity.com/paper/1711.07368/) (2017)

## Built on by (most cited first, 30 of 152)

- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2310.18961/) (2023)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2212.03588/) (2022)
- [Raising the Bar of AI-generated Image Detection with CLIP](https://mlsanity.com/paper/2312.00195/) (2023)
- [CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No](https://mlsanity.com/paper/2308.12213/) (2023)
- [LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning](https://mlsanity.com/paper/2306.01293/) (2023)
- [A Theoretical Study on Solving Continual Learning](https://mlsanity.com/paper/2211.02633/) (2022)
- [Dream the Impossible: Outlier Imagination with Diffusion Models](https://mlsanity.com/paper/2309.13415/) (2023)
- [Negative Label Guided OOD Detection with Pretrained Vision-Language Models](https://mlsanity.com/paper/2403.20078/) (2024)
- [FiLo: Zero-Shot Anomaly Detection by Fine-Grained Description and High-Quality Localization](https://mlsanity.com/paper/2404.13671/) (2024)
- [Learning Transferable Negative Prompts for Out-of-Distribution Detection](https://mlsanity.com/paper/2404.03248/) (2024)
- [ID-like Prompt Learning for Few-Shot Out-of-Distribution Detection](https://mlsanity.com/paper/2311.15243/) (2023)
- [GL-MCM: Global and Local Maximum Concept Matching for Zero-Shot Out-of-Distribution Detection](https://mlsanity.com/paper/2304.04521/) (2023)
- [Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2403.16005/) (2024)
- [Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey](https://mlsanity.com/paper/2407.21794/) (2024)
- [CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation](https://mlsanity.com/paper/2305.01040/) (2023)
- [Envisioning Outlier Exposure by Large Language Models for Out-of-Distribution Detection](https://mlsanity.com/paper/2406.00806/) (2024)
- [A framework for benchmarking class-out-of-distribution detection and its application to ImageNet](https://mlsanity.com/paper/2302.11893/) (2023)
- [PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection](https://mlsanity.com/paper/2410.00320/) (2024)
- [PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration](https://mlsanity.com/paper/2407.00203/) (2024)
- [Large Language Models for Anomaly and Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2409.01980/) (2024)
- [Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advances](https://mlsanity.com/paper/2409.11884/) (2024)
- [Building an Open-Vocabulary Video CLIP Model with Better Architectures, Optimization and Data](https://mlsanity.com/paper/2310.05010/) (2023)
- [Out-of-distribution Detection in Medical Image Analysis: A survey](https://mlsanity.com/paper/2404.18279/) (2024)
- [LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models](https://mlsanity.com/paper/2407.08966/) (2024)
- [SGDiff: A Style Guided Diffusion Model for Fashion Synthesis](https://mlsanity.com/paper/2308.07605/) (2023)
- [T-VSL: Text-Guided Visual Sound Source Localization in Mixtures](https://mlsanity.com/paper/2404.01751/) (2024)
- [AutoLabel: CLIP-based framework for Open-set Video Domain Adaptation](https://mlsanity.com/paper/2304.01110/) (2023)

## Abstract (arXiv)

In an out-of-distribution (OOD) detection problem, samples of known classes(also called in-distribution classes) are used to train a special classifier. In testing, the classifier can (1) classify the test samples of known classes to their respective classes and also (2) detect samples that do not belong to any of the known classes (i.e., they belong to some unknown or OOD classes). This paper studies the problem of zero-shot out-of-distribution(OOD) detection, which still performs the same two tasks in testing but has no training except using the given known class names. This paper proposes a novel yet simple method (called ZOC) to solve the problem. ZOC builds on top of the recent advances in zero-shot classification through multi-modal representation learning. It first extends the pre-trained language-vision model CLIP by training a text-based image description generator on top of CLIP. In testing, it uses the extended model to generate candidate unknown class names for each test sample and computes a confidence score based on both the known class names and candidate unknown class names for zero-shot OOD detection. Experimental results on 5 benchmark datasets for OOD detection demonstrate that ZOC outperforms the baselines by a large margin.