mlsanityData to 7 Oct 2026

GPT-4V-AD: Exploring Grounding Potential of VQA-oriented GPT-4V for Zero-shot Anomaly Detection

arXiv 2311.02612 ↗First posted 5 Nov 2023cs.CVcited by 23

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Large Multimodal Model (LMM) GPT-4V(ision) endows GPT-4 with visual grounding capabilities, making it possible to handle certain tasks through the Visual Question Answering (VQA) paradigm. This paper explores the potential of VQA-oriented GPT-4V in the recently popular visual Anomaly Detection (AD) and is the first to conduct qualitative and quantitative evaluations on the popular MVTec AD and VisA datasets. Considering that this task requires both image-/pixel-level evaluations, the proposed GPT-4V-AD framework contains three components: textbftextit1) Granular Region Division, textbftextit2) Prompt Designing, textbftextit3) Text2Segmentation for easy quantitative evaluation, and have made some different attempts for comparative analysis. The results show that GPT-4V can achieve certain results in the zero-shot AD task through a VQA paradigm, such as achieving image-level 77.1/88.0 and pixel-level 68.0/76.6 AU-ROCs on MVTec AD and VisA datasets, respectively. However, its performance still has a certain gap compared to the state-of-the-art zero-shot method, eg, WinCLIP and CLIP-AD, and further researches are needed. This study provides a baseline reference for the research of VQA-oriented LMM in the zero-shot AD task, and we also post several possible future works. Code is available at urlhttps://github.com/zhangzjn/GPT-4V-AD.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 42 references that are in the map. First posted on arXiv 5 Nov 2023.

Builds on

31 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models29 Aug

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Segment Anythingcited by 8,158
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2021Towards Total Recall in Industrial Anomaly Detectioncited by 772
  9. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  10. 2020PaDiM: a Patch Distribution Modeling Framework for Anomaly Detection and Localizationcited by 591
  11. 2021CutPaste: Self-Supervised Learning for Anomaly Detection and Localizationcited by 529
  12. 2021DRAEM -- A discriminatively trained reconstruction embedding for surface anomaly detectioncited by 510
18 more
  1. 2022Anomaly Detection via Reverse Distillation from One-Class Embeddingcited by 419
  2. 2022SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentationcited by 397
  3. 2019Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddingscited by 386
  4. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 311
  5. 2020Sub-Image Anomaly Detection with Deep Pyramid Correspondencescited by 297
  6. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  7. 2023AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detectioncited by 270
  8. 2022A Unified Model for Multi-class Anomaly Detectioncited by 267
  9. 2023SimpleNet: A Simple Network for Image Anomaly Detection and Localizationcited by 256
  10. 2020Same Same But DifferNet: Semi-Supervised Defect Detection with Normalizing Flowscited by 212
  11. 2023APRIL-GAN: A Zero-/Few-Shot Anomaly Classification and Segmentation Method for CVPR 2023 VAND Workshop Challenge Tracks 1&2: 1st Place on Zero-shot AD and 4th Place on Few-shot ADcited by 128
  12. 2021Natural Synthetic Anomalies for Self-Supervised Anomaly Detection and Localizationcited by 116
  13. 2022Omni-frequency Channel-selection Representations for Unsupervised Anomaly Detectioncited by 79
  14. 2023Pushing the Limits of Fewshot Anomaly Detection in Industry Vision: Graphcorecited by 75
  15. 2023Segment Any Anomaly without Training via Hybrid Prompt Regularizationcited by 59
  16. 2023CLIP-AD: A Language-Guided Staged Dual-Path Model for Zero-shot Anomaly Detectioncited by 58
  17. 2023Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluationcited by 46
  18. 2023An Early Evaluation of GPT-4V(ision)cited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 11; 2025: 7; 2026: 2.

23 papers.

Sort:

By area:

  1. 2023AnomalyDiffusion: Few-Shot Anomaly Image Generation with Diffusion Modelcited by 110
  2. 2024A Survey on Visual Anomaly Detection: Challenge, Approach, and Prospectcited by 69
  3. 2023Exploring Plain ViT Reconstruction for Multi-class Unsupervised Anomaly Detectioncited by 47
  4. 2024MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detectioncited by 46
  5. 2023DiAD: A Diffusion-based Framework for Multi-class Anomaly Detectioncited by 33
  6. 2024List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMscited by 32
  7. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  8. 2024M3DM-NR: RGB-3D Noisy-Resistant Industrial Anomaly Detection via Multimodal Denoisingcited by 24
  9. 2024Large Language Models for Anomaly and Out-of-Distribution Detection: A Surveycited by 24
  10. 2024LogiCode: an LLM-Driven Framework for Logical Anomaly Detectioncited by 18
  11. 2024Long-Tailed Anomaly Detection with Learnable Class Namescited by 14
  12. 2025UrbanSAM: Learning Invariance-Inspired Adapters for Segment Anything Models in Urban Constructioncited by 8
11 more
  1. 2025Foundation Models and Transformers for Anomaly Detection: A Surveycited by 8
  2. 2024CMOOD: Concept-based Multi-label OOD Detectioncited by 6
  3. 2024CableInspect-AD: An Expert-Annotated Anomaly Detection Datasetcited by 5
  4. 2025A Survey on Foundation-Model-Based Industrial Defect Detectioncited by 5
  5. 2026AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparisoncited by 5
  6. 2025InsightX Agent: An LMM-based Agentic Framework with Integrated Tools for Reliable X-ray NDT Analysiscited by 3
  7. 2025Toward Long-Tailed Online Anomaly Detection through Class-Agnostic Conceptscited by 1
  8. 2025IADGPT: Unified LVLM for Few-Shot Industrial Anomaly Detection, Localization, and Reasoning via In-Context Learningcited by 1
  9. 2026MMR-AD: A Large-Scale Multimodal Dataset for Benchmarking General Anomaly Detection with Multimodal Large Language Modelscited by 1
  10. 2024Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Modelscited by 0
  11. 2025Zero-Shot Anomaly Detection in Battery Thermal Images Using Visual Question Answering with Prior Knowledgecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023An Evaluation of GPT-4V and Gemini in Online VQA
  2. 2025Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models
  3. 2024Realizing Visual Question Answering for Education: GPT-4V as a Multimodal AI
  4. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V
  5. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering
  6. 2025How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks
  7. 2025GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generation
  8. 2025IAD-GPT: Advancing Visual Knowledge in Multimodal Large Language Model for Industrial Anomaly Detection