mlsanityData to 7 Oct 2026

LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models

arXiv 2407.08966 ↗First posted 12 Jul 2024cs.CV cs.AI cs.LGcited by 21

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Out-of-distribution (OOD) detection is crucial for model reliability, as it identifies samples from unknown classes and reduces errors due to unexpected inputs. Vision-Language Models (VLMs) such as CLIP are emerging as powerful tools for OOD detection by integrating multi-modal information. However, the practical application of such systems is challenged by manual prompt engineering, which demands domain expertise and is sensitive to linguistic nuances. In this paper, we introduce Label-driven Automated Prompt Tuning (LAPT), a novel approach to OOD detection that reduces the need for manual prompt engineering. We develop distribution-aware prompts with in-distribution (ID) class names and negative labels mined automatically. Training samples linked to these class labels are collected autonomously via image synthesis and retrieval methods, allowing for prompt learning without manual effort. We utilize a simple cross-entropy loss for prompt optimization, with cross-modal and cross-distribution mixing strategies to reduce image noise and explore the intermediate space between distributions, respectively. The LAPT framework operates autonomously, requiring only ID class names as input and eliminating the need for manual intervention. With extensive experiments, LAPT consistently outperforms manually crafted prompts, setting a new standard for OOD detection. Moreover, LAPT not only enhances the distinction between ID and OOD samples, but also improves the ID classification accuracy and strengthens the generalization robustness to covariate shifts, resulting in outstanding performance in challenging full-spectrum OOD detection tasks. Codes are available at urlhttps://github.com/YBZh/LAPT.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 74 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

59 of 74 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Negative Label Guided OOD Detection with Pretrained Vision-Language Models29 Mar
  2. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models26 Mar
  3. 2023How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?9 Jun
  4. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  9. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  10. 2013Describing Textures in the Wildcited by 2,110
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
38 more
  1. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  2. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  3. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  4. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  5. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  6. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  7. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  8. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  9. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
  10. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  11. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  12. 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
  13. 2022ViM: Out-Of-Distribution with Virtual-logit Matchingcited by 404
  14. 2021Open-Set Recognition: a Good Closed-Set Classifier is All You Need?cited by 351
  15. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 338
  16. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  17. 2022OpenOOD: Benchmarking Generalized Out-of-Distribution Detectioncited by 267
  18. 2021SSD: A Unified Framework for Self-Supervised Outlier Detectioncited by 264
  19. 2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 251
  20. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  21. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  22. 2022Extremely Simple Activation Shaping for Out-of-Distribution Detectioncited by 212
  23. 2021A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detectioncited by 206
  24. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 178
  25. 2021Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIPcited by 151
  26. 2021OpenGAN: Open-Set Recognition via Open Data Generationcited by 135
  27. 2023In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluationcited by 132
  28. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  29. 2023Non-Parametric Outlier Synthesiscited by 129
  30. 2022How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?cited by 128
  31. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  32. 2023Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentationcited by 100
  33. 2021Context-Aware Mixup for Domain Adaptive Semantic Segmentationcited by 79
  34. 2023Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancementcited by 77
  35. 2022Unknown-Aware Object Detection: Learning What You Don't Know from Videos in the Wildcited by 70
  36. 2021Mixture Outlier Exposure: Towards Out-of-Distribution Detection in Fine-grained Environmentscited by 65
  37. 2022Full-Spectrum Out-of-Distribution Detectioncited by 63
  38. 2021Energy-Based Open-World Uncertainty Modeling for Confidence Calibrationcited by 46

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 7.

21 papers.

Sort:

By area:

  1. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  2. 2024Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advancescited by 24
  3. 2024AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Modelscited by 16
  4. 2025Adaptive Multi-prompt Contrastive Network for Few-shot Out-of-distribution Detectioncited by 8
  5. 2025AdaSCALE: Adaptive Scaling for OOD Detectioncited by 7
  6. 2024AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?cited by 5
  7. 2025Recent Advances in Out-of-Distribution Detection with CLIP-Like Models: A Surveycited by 5
  8. 2025A Structured Review of Underwater Object Detection Challenges and Solutions: From Traditional to Large Vision Language Modelscited by 4
  9. 2025Mitigating the Modality Gap: Few-Shot Out-of-Distribution Detection with Multi-modal Prototypes and Image Bias Estimationcited by 3
  10. 2025Synthesizing Near-Boundary OOD Samples for Out-of-Distribution Detectioncited by 3
  11. 2025FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Modelscited by 2
  12. 2025NegRefine: Refining Negative Label-Based Zero-Shot OOD Detectioncited by 2
9 more
  1. 2025Delta Energy: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalizationcited by 1
  2. 2026Mind the Way You Select Negative Texts: Pursuing the Distance Consistency in OOD Detection with VLMscited by 1
  3. 2026Debiased Negative Mining Improves Out-of-distribution Detection with Pre-trained Vision-Language Modelscited by 1
  4. 2026Semantic Robustness Certification for Vision-Language Modelscited by 1
  5. 2026Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMscited by 1
  6. 2025OpenHAIV: A Framework Towards Practical Open-World Learningcited by 0
  7. 2026TINS: Test-time ID-prototype-separated Negative Semantics Learning for OOD Detectioncited by 0
  8. 2026Respecting Modality Gap in Post-hoc Out-of-distribution Detection with Pre-trained Vision-Language Modelscited by 0
  9. 2026MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Harnessing Large Language and Vision-Language Models for Robust Out-of-Distribution Detection
  2. 2025FA: Forced Prompt Learning of Vision-Language Models for Out-of-Distribution Detection
  3. 2025Out-of-Distribution Detection with Positive and Negative Prompt Supervision Using Large Language Models
  4. 2024Near OOD Detection for Vision-Language Prompt Learning with Contrastive Logit Score
  5. 2025An Empirical Analysis of VLM-based OOD Detection: Mechanisms, Advantages, and Sensitivity
  6. 2026TTL: Test-time Textual Learning for OOD Detection with Pretrained Vision-Language Models
  7. 2025SimLabel: Consistency-Guided OOD Detection with Pretrained Vision-Language Models
  8. 2024Self-Calibrated Tuning of Vision-Language Models for Out-of-Distribution Detection