mlsanityData to 7 Oct 2026

Reflexive Guidance: Improving OoDD in Vision-Language Models via Self-Guided Image-Adaptive Concept Generation

arXiv 2410.14975 ↗First posted 19 Oct 2024cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

With the recent emergence of foundation models trained on internet-scale data and demonstrating remarkable generalization capabilities, such foundation models have become more widely adopted, leading to an expanding range of application domains. Despite this rapid proliferation, the trustworthiness of foundation models remains underexplored. Specifically, the out-of-distribution detection (OoDD) capabilities of large vision-language models (LVLMs), such as GPT-4o, which are trained on massive multi-modal data, have not been sufficiently addressed. The disparity between their demonstrated potential and practical reliability raises concerns regarding the safe and trustworthy deployment of foundation models. To address this gap, we evaluate and analyze the OoDD capabilities of various proprietary and open-source LVLMs. Our investigation contributes to a better understanding of how these foundation models represent confidence scores through their generated natural language responses. Furthermore, we propose a self-guided prompting approach, termed Reflexive Guidance (ReGuide), aimed at enhancing the OoDD capability of LVLMs by leveraging self-generated image-adaptive concept suggestions. Experimental results demonstrate that our ReGuide enhances the performance of current LVLMs in both image classification and OoDD tasks. The lists of sampled images, along with the prompts and responses for each sample are available at https://github.com/daintlab/ReGuide.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 70 references that are in the map. First posted on arXiv 19 Oct 2024.

Builds on

55 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Negative Label Guided OOD Detection with Pretrained Vision-Language Models29 Mar

Other references · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  10. 2013Describing Textures in the Wildcited by 2,110
  11. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  12. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
38 more
  1. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  3. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  4. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  6. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  7. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
  8. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  9. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  10. 2022ViM: Out-Of-Distribution with Virtual-logit Matchingcited by 404
  11. 2017Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samplescited by 389
  12. 2021Open-Set Recognition: a Good Closed-Set Classifier is All You Need?cited by 351
  13. 2023Role-Play with Large Language Modelscited by 290
  14. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  15. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  16. 2022Extremely Simple Activation Shaping for Out-of-Distribution Detectioncited by 212
  17. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 178
  18. 2024Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Modelscited by 164
  19. 2021Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIPcited by 151
  20. 2023In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluationcited by 132
  21. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  22. 2023Non-Parametric Outlier Synthesiscited by 129
  23. 2020ATOM: Robustifying Out-of-distribution Detection Using Outlier Miningcited by 116
  24. 2020Confidence-Aware Learning for Deep Neural Networkscited by 113
  25. 2018Bias-Reduced Uncertainty Estimation for Deep Neural Classifierscited by 106
  26. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 93
  27. 2022Training OOD Detectors in their Natural Habitatscited by 86
  28. 2023Dream the Impossible: Outlier Imagination with Diffusion Modelscited by 77
  29. 2023Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancementcited by 77
  30. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  31. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 58
  32. 2024Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Modelscited by 51
  33. 2023Fast Decision Boundary based Out-of-Distribution Detectorcited by 48
  34. 2023Out-of-distribution Detection Learning with Unreliable Out-of-distribution Sourcescited by 31
  35. 2024Envisioning Outlier Exposure by Large Language Models for Out-of-Distribution Detectioncited by 30
  36. 2023DiffGuard: Semantic Mismatch-Guided Out-of-Distribution Detection using Pre-trained Diffusion Modelscited by 25
  37. 2021Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detectioncited by 23
  38. 2023How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigationcited by 22

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Outlier Synthesis via Hamiltonian Monte Carlo for Out-of-Distribution Detectioncited by 4
  2. 2026PWLR: Pairwise Witness Local Rejection for Boundary-Aware Out-of-Distribution Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025An Empirical Analysis of VLM-based OOD Detection: Mechanisms, Advantages, and Sensitivity
  2. 2023How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?
  3. 2026OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models
  4. 2024LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models
  5. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizability
  6. 2026Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models
  7. 2023Robust Fine-Tuning of Vision-Language Models for Domain Generalization
  8. 2024CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection