mlsanityData to 14 Oct 2026

Do LLMs Understand Visual Anomalies? Uncovering LLM's Capabilities in Zero-shot Anomaly Detection

arXiv 2404.09654 ↗First posted 15 Apr 2024cs.CV cs.MMcited by 36

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Large vision-language models (LVLMs) are markedly proficient in deriving visual representations guided by natural language. Recent explorations have utilized LVLMs to tackle zero-shot visual anomaly detection (VAD) challenges by pairing images with textual descriptions indicative of normal and abnormal conditions, referred to as anomaly prompts. However, existing approaches depend on static anomaly prompts that are prone to cross-semantic ambiguity, and prioritize global image-level representations over crucial local pixel-level image-to-text alignment that is necessary for accurate anomaly localization. In this paper, we present ALFA, a training-free approach designed to address these challenges via a unified model. We propose a run-time prompt adaptation strategy, which first generates informative anomaly prompts to leverage the capabilities of a large language model (LLM). This strategy is enhanced by a contextual scoring mechanism for per-image anomaly prompt adaptation and cross-semantic ambiguity mitigation. We further introduce a novel fine-grained aligner to fuse local pixel-level semantics for precise anomaly localization, by projecting the image-text alignment from global to local semantic spaces. Extensive evaluations on MVTec and VisA datasets confirm ALFA's effectiveness in harnessing the language potential for zero-shot VAD, achieving significant PRO improvements of 12.1% on MVTec and 8.9% on VisA compared to state-of-the-art approaches.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 67 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

39 of 67 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2023Segment Anythingcited by 8,234
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2021Towards Total Recall in Industrial Anomaly Detectioncited by 776
  11. 2020PaDiM: a Patch Distribution Modeling Framework for Anomaly Detection and Localizationcited by 594
  12. 2021Extract Free Dense Labels from CLIPcited by 583
27 more
  1. 2018Real-world Anomaly Detection in Surveillance Videoscited by 577
  2. 2019Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detectioncited by 564
  3. 2021CutPaste: Self-Supervised Learning for Anomaly Detection and Localizationcited by 531
  4. 2021DRAEM -- A discriminatively trained reconstruction embedding for surface anomaly detectioncited by 514
  5. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  6. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  7. 2020Multiresolution Knowledge Distillation for Anomaly Detectioncited by 285
  8. 2023AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detectioncited by 271
  9. 2022A Unified Model for Multi-class Anomaly Detectioncited by 268
  10. 2020Patch SVDD: Patch-level SVDD for Anomaly Detection and Segmentationcited by 212
  11. 2022DeSTSeg: Segmentation Guided Denoising Student-Teacher for Anomaly Detectioncited by 154
  12. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  13. 2023APRIL-GAN: A Zero-/Few-Shot Anomaly Classification and Segmentation Method for CVPR 2023 VAND Workshop Challenge Tracks 1&2: 1st Place on Zero-shot AD and 4th Place on Few-shot ADcited by 130
  14. 2021Self-Supervised Predictive Convolutional Attentive Block for Anomaly Detectioncited by 119
  15. 2024Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Promptscited by 89
  16. 2023Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Modelscited by 61
  17. 2023Multi-Modal Classifiers for Open-Vocabulary Object Detectioncited by 53
  18. 2020Encoding Structure-Texture Relation with P-Net for Anomaly Detection in Retinal Imagescited by 52
  19. 2024MuSc: Zero-Shot Industrial Anomaly Classification and Segmentation with Mutual Scoring of the Unlabeled Imagescited by 52
  20. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 51
  21. 2023EasyNet: An Easy Network for 3D Industrial Anomaly Detectioncited by 43
  22. 2020Any-Shot Sequential Anomaly Detection in Surveillance Videoscited by 34
  23. 2020MOCCA: Multi-Layer One-Class ClassificAtion for Anomaly Detectioncited by 31
  24. 2021ARM-Net: Adaptive Relation Modeling Network for Structured Datacited by 24
  25. 2023Random Word Data Augmentation with CLIP for Zero-Shot Anomaly Detectioncited by 16
  26. 2018Deep Generative Model using Unregularized Score for Anomaly Detection with Heterogeneous Complexitycited by 10
  27. 2023METER: A Dynamic Concept Adaptation Framework for Online Anomaly Detectioncited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 19; 2026: 9.

36 papers.

Sort:

By area:

  1. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 90
  2. 2025Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Modelscited by 54
  3. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  4. 2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
  5. 2024AD-LLM: Benchmarking Large Language Models for Anomaly Detectioncited by 26
  6. 2025Deep Learning Advancements in Anomaly Detection: A Comprehensive Surveycited by 17
  7. 2024VMAD: Visual-enhanced Multimodal Large Language Model for Zero-Shot Anomaly Detectioncited by 13
  8. 2024Detached and Interactive Multimodal Learningcited by 11
  9. 2025FiLo++: Zero-/Few-Shot Anomaly Detection by Fused Fine-Grained Descriptions and Deformable Localizationcited by 8
  10. 2025Foundation Models and Transformers for Anomaly Detection: A Surveycited by 8
  11. 2025A Survey on Foundation-Model-Based Industrial Defect Detectioncited by 5
  12. 2025AnomalyPainter: Vision-Language-Diffusion Synergy for Zero-Shot Realistic and Diverse Industrial Anomaly Synthesiscited by 5
24 more
  1. 2026AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparisoncited by 5
  2. 2024NC-NCD: Novel Class Discovery for Node Classificationcited by 4
  3. 2026AnomalyVFM -- Transforming Vision Foundation Models into Zero-Shot Anomaly Detectorscited by 4
  4. 2025Foundation Models for Anomaly Detection: Vision and Challengescited by 3
  5. 2025GIQ: Benchmarking 3D Geometric Reasoning of Vision Foundation Models with Simulated and Real Polyhedracited by 2
  6. 2025A Comprehensive Survey for Real-World Industrial Defect Detection: Challenges, Approaches, and Prospectscited by 2
  7. 2025ALARM: Automated MLLM-Based Anomaly Detection in Complex-EnviRonment Monitoring with Uncertainty Quantificationcited by 2
  8. 2025A Contrastive Learning-Guided Confident Meta-learning for Zero Shot Anomaly Detectioncited by 1
  9. 2025OmniAcc: Personalized Accessibility Assistant Using Generative AIcited by 1
  10. 2025Cerberus: Real-Time Video Anomaly Detection via Cascaded Vision-Language Modelscited by 1
  11. 2025The Forecast Critic: Leveraging Large Language Models for Poor Forecast Identificationcited by 1
  12. 2026Training Free Zero-Shot Visual Anomaly Localization via Diffusion Inversioncited by 1
  13. 2024Uncovering Capabilities of Model Pruning in Graph Contrastive Learningcited by 0
  14. 2025Intelligent IoT Attack Detection Design via ODLLM with Feature Ranking-based Knowledge Basecited by 0
  15. 2025PB-IAD: Utilizing multimodal foundation models for semantic industrial anomaly detection in dynamic manufacturing environmentscited by 0
  16. 2025An Improved Time Series Anomaly Detection by Applying Structural Similaritycited by 0
  17. 2025HyCoVAD: A Hybrid SSL-LLM Model for Complex Video Anomaly Detectioncited by 0
  18. 2025Noise & pattern: identity-anchored Tikhonov regularization for robust structural anomaly detectioncited by 0
  19. 2026Catching Every Ripple: Enhanced Anomaly Awareness via Dynamic Concept Adaptationcited by 0
  20. 2026Autonomous VR-Based Risk Detection for Situational Awareness in Dangerous Settingscited by 0
  21. 2026Hazard or Anomaly? Evaluating VLMs for Understanding Dangers and Discrepanciescited by 0
  22. 2026LogicTree-RAG: Logic Tree-guided Retrieval-Augmented Generation for Long-form Patent Draftingcited by 0
  23. 2026LLM4Trust: Exploring the Capabilities of Large Language Models for Trust Evaluationcited by 0
  24. 2026Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Bettingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Fine-Grained Vision-Language Alignment for Few-Shot Anomaly Detection
  2. 2026VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer
  3. 2026AG-VAS: Anchor-Guided Zero-Shot Visual Anomaly Segmentation with Large Multimodal Models
  4. 2025Beyond Human-prompting: Adaptive Prompt Tuning with Semantic Alignment for Anomaly Detection
  5. 2026NOVA: Normal-Side Modeling for Training-Free Zero-Shot Video Anomaly Detection
  6. 2024Bidirectional Multimodal Prompt Learning with Scale-Aware Training for Few-Shot Multi-Class Anomaly Detection
  7. 2024Quo Vadis, Anomaly Detection? LLMs and VLMs in the Spotlight
  8. 2026TIPS Over Tricks: Simple Prompts for Effective Zero-shot Anomaly Detection