# Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey

arXiv: https://arxiv.org/abs/2407.21794 · HTML: https://mlsanity.com/paper/2407.21794/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 181 of its 259 references that are in the map.
- First posted on arXiv 31 Jul 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 181)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)

## Built on by (most cited first, 30 of 36)

- [Process Reward Model with Q-Value Rankings](https://mlsanity.com/paper/2410.11287/) (2024)
- [Large Language Models for Anomaly and Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2409.01980/) (2024)
- [Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advances](https://mlsanity.com/paper/2409.11884/) (2024)
- [Self-Calibrated Tuning of Vision-Language Models for Out-of-Distribution Detection](https://mlsanity.com/paper/2411.03359/) (2024)
- [Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Paper](https://mlsanity.com/paper/2511.04583/) (2025)
- [Foundation Models and Transformers for Anomaly Detection: A Survey](https://mlsanity.com/paper/2507.15905/) (2025)
- [CMOOD: Concept-based Multi-label OOD Detection](https://mlsanity.com/paper/2411.13578/) (2024)
- [Recent Advances in Out-of-Distribution Detection with CLIP-Like Models: A Survey](https://mlsanity.com/paper/2505.02448/) (2025)
- [FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering](https://mlsanity.com/paper/2505.21755/) (2025)
- [Invariant Link Selector for Spatial-Temporal Out-of-Distribution Problem](https://mlsanity.com/paper/2505.24178/) (2025)
- [PhenoAssistant: A Conversational Multi-Agent AI System for Automated Plant Phenotyping](https://mlsanity.com/paper/2504.19818/) (2025)
- [Zero-Shot Scene Understanding for Automatic Target Recognition Using Large Vision-Language Models](https://mlsanity.com/paper/2501.07396/) (2025)
- [A Benchmark and Evaluation for Real-World Out-of-Distribution Detection Using Vision-Language Models](https://mlsanity.com/paper/2501.18463/) (2025)
- [Are vision language models robust to uncertain inputs?](https://mlsanity.com/paper/2505.11804/) (2025)
- [Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracy](https://mlsanity.com/paper/2509.21173/) (2025)
- [VLMDiff: Leveraging Vision-Language Models for Multi-Class Anomaly Detection with Diffusion](https://mlsanity.com/paper/2511.08173/) (2025)
- [Activation Matters: Test-time Activated Negative Labels for OOD Detection with Vision-Language Models](https://mlsanity.com/paper/2603.25250/) (2026)
- [The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detection](https://mlsanity.com/paper/2606.21579/) (2026)
- [Action Hints: Semantic Typicality and Context Uniqueness for Generalizable Skeleton-based Video Anomaly Detection](https://mlsanity.com/paper/2509.11058/) (2025)
- [Delta Energy: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization](https://mlsanity.com/paper/2510.11296/) (2025)
- [Sparse, self-organizing ensembles of local kernels detect rare statistical anomalies](https://mlsanity.com/paper/2511.03095/) (2025)
- [ICONIC-444: A 3.1-Million-Image Dataset for OOD Detection Research](https://mlsanity.com/paper/2601.10802/) (2026)
- [Mitigating Simplicity Bias in OOD Detection through Object Co-occurrence Analysis](https://mlsanity.com/paper/2605.07821/) (2026)
- [How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey](https://mlsanity.com/paper/2412.08158/) (2024)
- [Harnessing Large Language and Vision-Language Models for Robust Out-of-Distribution Detection](https://mlsanity.com/paper/2501.05228/) (2025)
- [Explaining Low Perception Model Competency with High-Competency Counterfactuals](https://mlsanity.com/paper/2504.05254/) (2025)
- [FindMeIfYouCan: Bringing Open Set metrics to textitnear ,  textitfar  and textitfarther Out-of-Distribution Object Detection](https://mlsanity.com/paper/2506.14008/) (2025)
- [Fake or Real, Can Robots Tell? Evaluating VLM Robustness to Domain Shift in Single-View Robotic Scene Understanding](https://mlsanity.com/paper/2506.19579/) (2025)
- [COOkeD: Ensemble-based OOD detection in the era of zero-shot CLIP](https://mlsanity.com/paper/2507.22576/) (2025)
- [CLAIR: CLIP-Aided Weakly Supervised Zero-Shot Cross-Domain Image Retrieval](https://mlsanity.com/paper/2508.12290/) (2025)

## Abstract (arXiv)

Detecting out-of-distribution (OOD) samples is crucial for ensuring the safety of machine learning systems and has shaped the field of OOD detection. Meanwhile, several other problems are closely related to OOD detection, including anomaly detection (AD), novelty detection (ND), open set recognition (OSR), and outlier detection (OD). To unify these problems, a generalized OOD detection framework was proposed, taxonomically categorizing these five problems. However, Vision Language Models (VLMs) such as CLIP have significantly changed the paradigm and blurred the boundaries between these fields, again confusing researchers. In this survey, we first present a generalized OOD detection v2, encapsulating the evolution of these fields in the VLM era. Our framework reveals that, with some field inactivity and integration, the demanding challenges have become OOD detection and AD. Then, we highlight the significant shift in the definition, problem settings, and benchmarks; we thus feature a comprehensive review of the methodology for OOD detection and related tasks to clarify their relationship to OOD detection. Finally, we explore the advancements in the emerging Large Vision Language Model (LVLM) era, such as GPT-4V. We conclude with open challenges and future directions. The resource is available at https://github.com/AtsuMiyai/Awesome-OOD-VLM.