mlsanityData to 7 Oct 2026

Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey

arXiv 2407.21794 ↗First posted 31 Jul 2024cs.CV cs.AI cs.LGcited by 36

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Detecting out-of-distribution (OOD) samples is crucial for ensuring the safety of machine learning systems and has shaped the field of OOD detection. Meanwhile, several other problems are closely related to OOD detection, including anomaly detection (AD), novelty detection (ND), open set recognition (OSR), and outlier detection (OD). To unify these problems, a generalized OOD detection framework was proposed, taxonomically categorizing these five problems. However, Vision Language Models (VLMs) such as CLIP have significantly changed the paradigm and blurred the boundaries between these fields, again confusing researchers. In this survey, we first present a generalized OOD detection v2, encapsulating the evolution of these fields in the VLM era. Our framework reveals that, with some field inactivity and integration, the demanding challenges have become OOD detection and AD. Then, we highlight the significant shift in the definition, problem settings, and benchmarks; we thus feature a comprehensive review of the methodology for OOD detection and related tasks to clarify their relationship to OOD detection. Finally, we explore the advancements in the emerging Large Vision Language Model (LVLM) era, such as GPT-4V. We conclude with open challenges and future directions. The resource is available at https://github.com/AtsuMiyai/Awesome-OOD-VLM.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 181 of its 259 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

181 of 259 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Category-Extensible Out-of-Distribution Detection via Hierarchical Context Descriptions23 Jul
  2. 2024LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models12 Jul
  3. 2024GalLoP: Learning Global and Local Prompts for Vision-Language Models1 Jul
  4. 2024DeCoOp: Robust Prompt Tuning with Out-of-Distribution Detection1 Jun
  5. 2024Near OOD Detection for Vision-Language Prompt Learning with Contrastive Logit Score25 May
  6. 2024PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization13 Apr
  7. 2024Negative Label Guided OOD Detection with Pretrained Vision-Language Models29 Mar
  8. 2023Realistic Unsupervised CLIP Fine-tuning with Universal Entropy Optimization24 Aug
  9. 2023How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?9 Jun
  10. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentation23 Feb
  11. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Models28 Nov
  12. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
5 more
  1. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  2. 2022Prompt Distribution Learning6 May
  3. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 164

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  12. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  10. 2013Describing Textures in the Wildcited by 2,110
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  13. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  14. 2022Visual Prompt Tuningcited by 1,674
  15. 2016Concrete Problems in AI Safetycited by 1,582
  16. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  17. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  18. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  19. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  20. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  21. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  22. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  23. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  24. 2021Towards Total Recall in Industrial Anomaly Detectioncited by 772
  25. 2021Domain Generalization: A Surveycited by 760
  26. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  27. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  28. 2021ImageNet-21K Pretraining for the Massescited by 607
  29. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  30. 2020PaDiM: a Patch Distribution Modeling Framework for Anomaly Detection and Localizationcited by 591
  31. 2018Real-world Anomaly Detection in Surveillance Videoscited by 570
  32. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  33. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  34. 2021CutPaste: Self-Supervised Learning for Anomaly Detection and Localizationcited by 529
  35. 2021DRAEM -- A discriminatively trained reconstruction embedding for surface anomaly detectioncited by 510
  36. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  37. 2020A Unifying Review of Deep and Shallow Anomaly Detectioncited by 423
  38. 2022ViM: Out-Of-Distribution with Virtual-logit Matchingcited by 404

50 most cited of 164 All 164 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 20; 2026: 10.

36 papers.

Sort:

By area:

  1. 2024Process Reward Model with Q-Value Rankingscited by 78
  2. 2024Large Language Models for Anomaly and Out-of-Distribution Detection: A Surveycited by 24
  3. 2024Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advancescited by 24
  4. 2024Self-Calibrated Tuning of Vision-Language Models for Out-of-Distribution Detectioncited by 15
  5. 2025Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Papercited by 10
  6. 2025Foundation Models and Transformers for Anomaly Detection: A Surveycited by 8
  7. 2024CMOOD: Concept-based Multi-label OOD Detectioncited by 6
  8. 2025Recent Advances in Out-of-Distribution Detection with CLIP-Like Models: A Surveycited by 5
  9. 2025FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answeringcited by 5
  10. 2025Invariant Link Selector for Spatial-Temporal Out-of-Distribution Problemcited by 5
  11. 2025PhenoAssistant: A Conversational Multi-Agent AI System for Automated Plant Phenotypingcited by 4
  12. 2025Zero-Shot Scene Understanding for Automatic Target Recognition Using Large Vision-Language Modelscited by 3
24 more
  1. 2025A Benchmark and Evaluation for Real-World Out-of-Distribution Detection Using Vision-Language Modelscited by 3
  2. 2025Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracycited by 3
  3. 2025VLMDiff: Leveraging Vision-Language Models for Multi-Class Anomaly Detection with Diffusioncited by 3
  4. 2025Are vision language models robust to uncertain inputs?cited by 2
  5. 2026Activation Matters: Test-time Activated Negative Labels for OOD Detection with Vision-Language Modelscited by 2
  6. 2026The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detectioncited by 2
  7. 2025Action Hints: Semantic Typicality and Context Uniqueness for Generalizable Skeleton-based Video Anomaly Detectioncited by 1
  8. 2025Delta Energy: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalizationcited by 1
  9. 2025Sparse, self-organizing ensembles of local kernels detect rare statistical anomaliescited by 1
  10. 2026ICONIC-444: A 3.1-Million-Image Dataset for OOD Detection Researchcited by 1
  11. 2026Mitigating Simplicity Bias in OOD Detection through Object Co-occurrence Analysiscited by 1
  12. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Surveycited by 0
  13. 2025Harnessing Large Language and Vision-Language Models for Robust Out-of-Distribution Detectioncited by 0
  14. 2025Explaining Low Perception Model Competency with High-Competency Counterfactualscited by 0
  15. 2025FindMeIfYouCan: Bringing Open Set metrics to textitnear , textitfar and textitfarther Out-of-Distribution Object Detectioncited by 0
  16. 2025Fake or Real, Can Robots Tell? Evaluating VLM Robustness to Domain Shift in Single-View Robotic Scene Understandingcited by 0
  17. 2025COOkeD: Ensemble-based OOD detection in the era of zero-shot CLIPcited by 0
  18. 2025CLAIR: CLIP-Aided Weakly Supervised Zero-Shot Cross-Domain Image Retrievalcited by 0
  19. 2026Out-of-Distribution Object Detection in Street Scenes via Synthetic Outlier Exposure and Transfer Learningcited by 0
  20. 2026Bounding Box Anomaly Scoring for simple and efficient Out-of-Distribution detectioncited by 0
  21. 2026Ranked Activation Shift for Post-Hoc Out-of-Distribution Detectioncited by 0
  22. 2026From Local Geometry to Global Pseudo Labeling for Robust Positive Unlabeled Learning under Covariate Shiftcited by 0
  23. 2026GeoMAD: Geometry-Aware Multi-View Anomaly Detection via Deformable Fusion and Distributional Alignmentcited by 0
  24. 2026Region-Level Black-Box Defense Against Stealthy Embedding-Space Backdoors in CLIPcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Vision Also You Need: Navigating Out-of-Distribution Detection with Multimodal Large Language Model
  2. 2025Delving into Out-of-Distribution Detection with Medical Vision-Language Models
  3. 2025Out-of-Distribution Detection with Positive and Negative Prompt Supervision Using Large Language Models
  4. 2025An Empirical Analysis of VLM-based OOD Detection: Mechanisms, Advantages, and Sensitivity
  5. 2025FA: Forced Prompt Learning of Vision-Language Models for Out-of-Distribution Detection
  6. 2025SimLabel: Consistency-Guided OOD Detection with Pretrained Vision-Language Models
  7. 2024Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detection
  8. 2026Debiased Negative Mining Improves Out-of-distribution Detection with Pre-trained Vision-Language Models