mlsanityData to 7 Oct 2026

Delving into Out-of-Distribution Detection with Vision-Language Representations

arXiv 2211.13445 ↗First posted 24 Nov 2022cs.CV cs.AI cs.LGcited by 215

Not on the map. Why?

Cited by 67 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recognizing out-of-distribution (OOD) samples is critical for machine learning systems deployed in the open world. The vast majority of OOD detection methods are driven by a single modality (e.g., either vision or language), leaving the rich information in multi-modal representations untapped. Inspired by the recent success of vision-language pre-training, this paper enriches the landscape of OOD detection from a single-modal to a multi-modal regime. Particularly, we propose Maximum Concept Matching (MCM), a simple yet effective zero-shot OOD detection method based on aligning visual features with textual concepts. We contribute in-depth analysis and theoretical insights to understand the effectiveness of MCM. Extensive experiments demonstrate that MCM achieves superior performance on a wide variety of real-world tasks. MCM with vision-language features outperforms a common baseline with pure visual features on a hard OOD task with semantically similar classes by 13.1% (AUROC). Code is available at https://github.com/deeplearning-wisc/MCM.

Key facts. Cited by 215 papers in the map (data through 7 Oct 2026). Builds on 78 of its 112 references that are in the map. First posted on arXiv 24 Nov 2022.

Builds on

78 of 112 references.

References · 78

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  10. 2013Describing Textures in the Wildcited by 2,110
  11. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  12. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
38 more
  1. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  2. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  3. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  4. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  5. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  8. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  9. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  10. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  11. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  12. 2018Recognition in Terra Incognitacited by 655
  13. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  14. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  15. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  16. 2015Towards Open Set Deep Networkscited by 533
  17. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  18. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 451
  19. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  20. 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
  21. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  22. 2020Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Datacited by 408
  23. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  24. 2020Understanding the Behaviour of Contrastive Losscited by 397
  25. 2021Open-Set Recognition: a Good Closed-Set Classifier is All You Need?cited by 351
  26. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
  27. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 338
  28. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  29. 2022VOS: Learning What You Don't Know by Virtual Outlier Synthesiscited by 300
  30. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  31. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  32. 2021SSD: A Unified Framework for Self-Supervised Outlier Detectioncited by 264
  33. 2018Learning Confidence for Out-of-Distribution Detection in Neural Networkscited by 245
  34. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  35. 2020Why Normalizing Flows Fail to Detect Out-of-Distribution Datacited by 220
  36. 2019Input complexity and out-of-distribution detection with likelihood-based generative modelscited by 186
  37. 2020Contrastive Training for Improved Out-of-Distribution Detectioncited by 186
  38. 2021DICE: Leveraging Sparsification for Out-of-Distribution Detectioncited by 166

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 41; 2024: 75; 2025: 55; 2026: 44.

215 papers.

Sort:

By area:

  1. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 178
  2. 2023Raising the Bar of AI-generated Image Detection with CLIPcited by 163
  3. 2023In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluationcited by 132
  4. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  5. 2023Non-Parametric Outlier Synthesiscited by 129
  6. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  7. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  8. 2024Process Reward Model with Q-Value Rankingscited by 78
  9. 2023Dream the Impossible: Outlier Imagination with Diffusion Modelscited by 77
  10. 2024Negative Label Guided OOD Detection with Pretrained Vision-Language Modelscited by 66
  11. 2024A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)cited by 44
  12. 2024Learning Transferable Negative Prompts for Out-of-Distribution Detectioncited by 43
38 more
  1. 2023ID-like Prompt Learning for Few-Shot Out-of-Distribution Detectioncited by 42
  2. 2023How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?cited by 38
  3. 2023Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Drivingcited by 37
  4. 2023GL-MCM: Global and Local Maximum Concept Matching for Zero-Shot Out-of-Distribution Detectioncited by 36
  5. 2024How Does Unlabeled Data Provably Help Out-of-Distribution Detection?cited by 36
  6. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  7. 2023Detecting Out-of-Distribution Through the Lens of Neural Collapsecited by 33
  8. 2023A Survey on Out-of-Distribution Detection in NLPcited by 31
  9. 2024Envisioning Outlier Exposure by Large Language Models for Out-of-Distribution Detectioncited by 30
  10. 2025Mahalanobis++: Improving OOD Detection via Feature Normalizationcited by 30
  11. 2023Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detectioncited by 29
  12. 2024GalLoP: Learning Global and Local Prompts for Vision-Language Modelscited by 29
  13. 2024PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detectioncited by 29
  14. 2024MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalitiescited by 28
  15. 2024Large Language Models for Anomaly and Out-of-Distribution Detection: A Surveycited by 24
  16. 2024Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advancescited by 24
  17. 2024ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Objectcited by 22
  18. 2023When and How Does Known Class Help Discover Unknown Ones? Provable Understanding Through Spectral Analysiscited by 21
  19. 2023Building an Open-Vocabulary Video CLIP Model with Better Architectures, Optimization and Datacited by 21
  20. 2024Zero-shot Object Counting with Good Exemplarscited by 21
  21. 2024LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Modelscited by 21
  22. 2024Unknown Prompt, the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalizationcited by 20
  23. 2024Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwardscited by 19
  24. 2023From Global to Local: Multi-scale Out-of-distribution Detectioncited by 18
  25. 2024Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalizationcited by 17
  26. 2023DOS: Diverse Outlier Sampling for Out-of-Distribution Detectioncited by 16
  27. 2023Unleashing Mask: Explore the Intrinsic Out-of-Distribution Detection Capabilitycited by 16
  28. 2024AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Modelscited by 16
  29. 2024Learning Structured Representations with Hyperbolic Embeddingscited by 16
  30. 2025OODD: Test-time Out-of-Distribution Detection with Dynamic Dictionarycited by 16
  31. 2025Reliable Few-shot Learning under Dual Noisescited by 16
  32. 2024Self-Calibrated Tuning of Vision-Language Models for Out-of-Distribution Detectioncited by 15
  33. 2024Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidancecited by 14
  34. 2024Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detectioncited by 14
  35. 2023Unified Out-Of-Distribution Detection: A Model-Specific Perspectivecited by 13
  36. 2023Adversarial Attacks on Foundational Vision Modelscited by 13
  37. 2024Gradient-Regularized Out-of-Distribution Detectioncited by 13
  38. 2024CLIPScope: Enhancing Zero-Shot OOD Detection with Bayesian Scoringcited by 13

50 most cited of 215 All 215 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Harnessing Large Language and Vision-Language Models for Robust Out-of-Distribution Detection
  2. 2025Out-of-Distribution Detection with Positive and Negative Prompt Supervision Using Large Language Models
  3. 2025Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal Representations
  4. 2024CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection
  5. 2023Class Relevance Learning For Out-of-distribution Detection
  6. 2024Can OOD Object Detectors Learn from Foundation Models?
  7. 2025ODOV: Benchmark the Open-Domain Open-Vocabulary Object Detection
  8. 2025ANTS: Adaptive Negative Textual Space Shaping for OOD Detection via Test-Time MLLM Understanding and Reasoning