mlsanityData to 14 Oct 2026

Towards Generic Anomaly Detection and Understanding: Large-scale Visual-linguistic Model (GPT-4V) Takes the Lead

arXiv 2311.02782 ↗First posted 5 Nov 2023cs.CV cs.AIcited by 40

Not on the map. Why?

Abstract (arXiv)

Anomaly detection is a crucial task across different domains and data types. However, existing anomaly detection models are often designed for specific domains and modalities. This study explores the use of GPT-4V(ision), a powerful visual-linguistic model, to address anomaly detection tasks in a generic manner. We investigate the application of GPT-4V in multi-modality, multi-domain anomaly detection tasks, including image, video, point cloud, and time series data, across multiple application areas, such as industrial, medical, logical, video, 3D anomaly detection, and localization tasks. To enhance GPT-4V's performance, we incorporate different kinds of additional cues such as class information, human expertise, and reference images as prompts.Based on our experiments, GPT-4V proves to be highly effective in detecting and explaining global and fine-grained semantic patterns in zero/one-shot anomaly detection. This enables accurate differentiation between normal and abnormal instances. Although we conducted extensive evaluations in this study, there is still room for future evaluation to further exploit GPT-4V's generic anomaly detection capacity from different aspects. These include exploring quantitative metrics, expanding evaluation benchmarks, incorporating multi-round interactions, and incorporating human feedback loops. Nevertheless, GPT-4V exhibits promising performance in generic anomaly detection and understanding, thus opening up a new avenue for anomaly detection.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 33 of its 116 references that are in the map. First posted on arXiv 5 Nov 2023.

Builds on

33 of 116 references.

References · 33

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2023PaLM 2 Technical Reportcited by 1,162
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  7. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  8. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  9. 2019Deep Learning for Anomaly Detection: A Surveycited by 358
  10. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 315
  11. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  12. 2023AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detectioncited by 271
21 more
  1. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  2. 2023AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Modelscited by 171
  3. 2023EfficientAD: Accurate Visual Anomaly Detection at Millisecond-Level Latenciescited by 160
  4. 2023Multimodal Industrial Anomaly Detection via Hybrid Fusioncited by 139
  5. 2023APRIL-GAN: A Zero-/Few-Shot Anomaly Classification and Segmentation Method for CVPR 2023 VAND Workshop Challenge Tracks 1&2: 1st Place on Zero-shot AD and 4th Place on Few-shot ADcited by 130
  6. 2023Real3D-AD: A Dataset of Point Cloud Anomaly Detectioncited by 87
  7. 2022Omni-frequency Channel-selection Representations for Unsupervised Anomaly Detectioncited by 80
  8. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  9. 2023Complementary Pseudo Multimodal Feature for Point Cloud Anomaly Detectioncited by 75
  10. 2022Anomaly Detection in 3D Point Clouds using Deep Geometric Descriptorscited by 66
  11. 2023Hierarchical Vector Quantized Transformer for Multi-class Unsupervised Anomaly Detectioncited by 66
  12. 2023IM-IAD: Industrial Image Anomaly Detection Benchmark in Manufacturingcited by 64
  13. 2023Segment Any Anomaly without Training via Hybrid Prompt Regularizationcited by 60
  14. 2020RobustTAD: Robust Time Series Anomaly Detection via Decomposition and Convolutional Neural Networkscited by 55
  15. 2023PAD: A Dataset and Benchmark for Pose-agnostic Anomaly Detectioncited by 46
  16. 2020Viral Pneumonia Screening on Chest X-ray Images Using Confidence-Aware Anomaly Detectioncited by 28
  17. 2016Anomaly Detection in Video Using Predictive Convolutional Long Short-Term Memory Networkscited by 24
  18. 2023Component-aware anomaly detection framework for adjustable and logical industrial visual inspectioncited by 22
  19. 2023Contextual Affinity Distillation for Image Anomaly Detectioncited by 18
  20. 2023COVID-VTS: Fact Extraction and Verification on Short Video Platformscited by 15
  21. 2022Composite Convolution: a Flexible Operator for Deep Learning on 3D Point Cloudscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 20; 2025: 9; 2026: 5.

40 papers.

Sort:

By area:

  1. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 150
  2. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  3. 2024A Survey on Visual Anomaly Detection: Challenge, Approach, and Prospectcited by 69
  4. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 57
  5. 2024Follow the Rules: Reasoning for Video Anomaly Detection with Large Language Modelscited by 57
  6. 2024MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detectioncited by 47
  7. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  8. 2023GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?cited by 35
  9. 2024Large Language Models for Anomaly and Out-of-Distribution Detection: A Surveycited by 27
  10. 2024AD-LLM: Benchmarking Large Language Models for Anomaly Detectioncited by 26
  11. 2023Chain of Visual Perception: Harnessing Multimodal Large Language Models for Zero-shot Camouflaged Object Detectioncited by 22
  12. 2023How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigationcited by 22
28 more
  1. 2024LogiCode: an LLM-Driven Framework for Logical Anomaly Detectioncited by 19
  2. 2023Large Foundation Models for Power Systemscited by 18
  3. 2024Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Modelscited by 18
  4. 2025OmniAD: Detect and Understand Industrial Anomaly via Multimodal Reasoningcited by 18
  5. 2024VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigationcited by 14
  6. 2024Long-Tailed Anomaly Detection with Learnable Class Namescited by 14
  7. 2024PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generationcited by 13
  8. 2025EIAD: Explainable Industrial Anomaly Detection Via Multi-Modal Large Language Modelscited by 11
  9. 2024MFE-ETP: A Comprehensive Evaluation Benchmark for Multi-modal Foundation Models on Embodied Task Planningcited by 8
  10. 2025Foundation Models and Transformers for Anomaly Detection: A Surveycited by 8
  11. 2025AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimizationcited by 8
  12. 2024GlanceVAD: Exploring Glance Supervision for Label-efficient Video Anomaly Detectioncited by 7
  13. 2023GPT-4V(ision) as A Social Media Analysis Enginecited by 6
  14. 2024CMOOD: Concept-based Multi-label OOD Detectioncited by 6
  15. 20253DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4ocited by 6
  16. 2024NPHardEval4V: Dynamic Evaluation of Large Vision-Language Models with Effects of Visioncited by 5
  17. 2024CableInspect-AD: An Expert-Annotated Anomaly Detection Datasetcited by 5
  18. 2025A Survey on Foundation-Model-Based Industrial Defect Detectioncited by 5
  19. 2026AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparisoncited by 5
  20. 2025Exploring Few-Shot Defect Segmentation in General Industrial Scenarios with Metric Learning and Vision Foundation Modelscited by 4
  21. 2025Examining the Source of Defects from a Mechanical Perspective for 3D Anomaly Detectioncited by 4
  22. 2025InsightX Agent: An LMM-based Agentic Framework with Integrated Tools for Reliable X-ray NDT Analysiscited by 3
  23. 2026MMR-AD: A Large-Scale Multimodal Dataset for Benchmarking General Anomaly Detection with Multimodal Large Language Modelscited by 1
  24. 2024Generative Denoise Distillation: Simple Stochastic Noises Induce Efficient Knowledge Transfer for Dense Predictioncited by 0
  25. 2024Investigating the Semantic Robustness of CLIP-based Zero-Shot Anomaly Segmentationcited by 0
  26. 2026Open-Set Supervised 3D Anomaly Detection: An Industrial Dataset and a Generalisable Framework for Unknown Defectscited by 0
  27. 2026Fine-Tuning Vision-Language Models for Understanding Current Damage and Scoring Priority with Quality Guard Agentcited by 0
  28. 2026RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistantscited by 0