mlsanityData to 14 Oct 2026

VL4AD: Vision-Language Models Improve Pixel-wise Anomaly Detection

arXiv 2409.17330 ↗First posted 25 Sep 2024cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Semantic segmentation networks have achieved significant success under the assumption of independent and identically distributed data. However, these networks often struggle to detect anomalies from unknown semantic classes due to the limited set of visual concepts they are typically trained on. To address this issue, anomaly segmentation often involves fine-tuning on outlier samples, necessitating additional efforts for data collection, labeling, and model retraining. Seeking to avoid this cumbersome work, we take a different approach and propose to incorporate Vision-Language (VL) encoders into existing anomaly detectors to leverage the semantically broad VL pre-training for improved outlier awareness. Additionally, we propose a new scoring function that enables data- and training-free outlier supervision via textual prompts. The resulting VL4AD model, which includes max-logit prompt ensembling and a class-merging strategy, achieves competitive performance on widely used benchmark datasets, thereby demonstrating the potential of vision-language models for pixel-wise anomaly detection.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 57 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

52 of 57 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Anomaly Detection by Adapting a pre-trained Vision Language Model14 Mar
  2. 2023AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models29 Aug

Other references · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2017Mask R-CNNcited by 7,277
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2022A ConvNet for the 2020scited by 3,579
38 more
  1. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  2. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  4. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  5. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  6. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  7. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  8. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
  9. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 600
  10. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  11. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  12. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  13. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  14. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  15. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  16. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 216
  17. 2020Patch SVDD: Patch-level SVDD for Anomaly Detection and Segmentationcited by 212
  18. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
  19. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 179
  20. 2021Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIPcited by 152
  21. 2021SegmentMeIfYouCan: A Benchmark for Anomaly Segmentationcited by 150
  22. 2019Detecting the Unexpected via Image Resynthesiscited by 137
  23. 2023In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluationcited by 134
  24. 2020Entropy Maximization and Meta Classification for Out-Of-Distribution Detection in Semantic Segmentationcited by 133
  25. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  26. 2019The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentationcited by 122
  27. 2021Pixel-wise Anomaly Detection in Complex Driving Scenescited by 109
  28. 2021Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenescited by 83
  29. 2022RbA: Segmenting Unknown Regions Rejected by Allcited by 67
  30. 2018Prediction Error Meta Classification in Semantic Segmentation: Detection via Aggregated Dispersion Measures of Softmax Probabilitiescited by 54
  31. 2022DenseHybrid: Hybrid Anomaly Detection for Dense Open-set Recognitioncited by 42
  32. 2023On Advantages of Mask-level Recognition for Outlier-aware Segmentationcited by 42
  33. 2021Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentationcited by 40
  34. 2023Maskomaly:Zero-Shot Mask Anomaly Segmentationcited by 34
  35. 2024Outlier detection by ensembling uncertainty with negative objectnesscited by 34
  36. 2020Detecting Road Obstacles by Erasing Themcited by 29
  37. 2023Strong but simple: A Baseline for Domain Generalized Dense Perception by CLIP-based Transfer Learningcited by 20
  38. 2023Three ways to improve feature alignment for open vocabulary detectioncited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026LARAD: Layout-Aware Road Anomaly Detection via Spatial-Logic Reasoningcited by 0
  2. 2026RoG-DAgger: Rollout-Guided Post-Training for End-to-End Drivingcited by 0
  3. 2026Combining Foundation Model Confidence and Monocular Depth for Training-Free Out-of-Distribution Segmentationcited by 0
  4. 2026SAFe: Segment-guided Aggregation of Feature Densities for Anomaly-aware Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Analyzing VLM-Based Approaches for Anomaly Classification and Segmentation
  2. 2024Vision-Language Models Assisted Unsupervised Video Anomaly Detection
  3. 2026Vision-Language Feature Alignment for Road Anomaly Segmentation
  4. 2026One Language-Free Foundation Model Is Enough for Universal Vision Anomaly Detection
  5. 2024Do LLMs Understand Visual Anomalies? Uncovering LLM's Capabilities in Zero-shot Anomaly Detection
  6. 2025VLMDiff: Leveraging Vision-Language Models for Multi-Class Anomaly Detection with Diffusion
  7. 2025From Benchmarks to Reality: Advancing Visual Anomaly Detection by the VAND 3.0 Challenge
  8. 2026H2VLR: Heterogeneous Hypergraph Vision-Language Reasoning for Few-Shot Anomaly Detection