# AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models

arXiv: https://arxiv.org/abs/2308.15366 · HTML: https://mlsanity.com/paper/2308.15366/

## Key facts

- Cited by 171 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 29 Aug 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Towards Total Recall in Industrial Anomaly Detection](https://mlsanity.com/paper/2106.08265/) (2021)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [PaDiM: a Patch Distribution Modeling Framework for Anomaly Detection and Localization](https://mlsanity.com/paper/2011.08785/) (2020)
- [CutPaste: Self-Supervised Learning for Anomaly Detection and Localization](https://mlsanity.com/paper/2104.04015/) (2021)
- [VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks](https://mlsanity.com/paper/2305.11175/) (2023)
- [SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2207.14315/) (2022)
- [PandaGPT: One Model To Instruction-Follow Them All](https://mlsanity.com/paper/2305.16355/) (2023)
- [CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows](https://mlsanity.com/paper/2107.12571/) (2021)
- [WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation](https://mlsanity.com/paper/2303.14814/) (2023)
- [Sub-Image Anomaly Detection with Deep Pyramid Correspondences](https://mlsanity.com/paper/2005.02357/) (2020)
- [A Unified Model for Multi-class Anomaly Detection](https://mlsanity.com/paper/2206.03687/) (2022)
- [Patch SVDD: Patch-level SVDD for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2006.16067/) (2020)
- [CFA: Coupled-hypersphere-based Feature Adaptation for Target-Oriented Anomaly Localization](https://mlsanity.com/paper/2206.04325/) (2022)
- [Registration based Few-Shot Anomaly Detection](https://mlsanity.com/paper/2207.07361/) (2022)
- [APRIL-GAN: A Zero-/Few-Shot Anomaly Classification and Segmentation Method for CVPR 2023 VAND Workshop Challenge Tracks 1&2: 1st Place on Zero-shot AD and 4th Place on Few-shot AD](https://mlsanity.com/paper/2305.17382/) (2023)
- [Natural Synthetic Anomalies for Self-Supervised Anomaly Detection and Localization](https://mlsanity.com/paper/2109.15222/) (2021)
- [Inpainting Transformer for Anomaly Detection](https://mlsanity.com/paper/2104.13897/) (2021)
- [Pushing the Limits of Fewshot Anomaly Detection in Industry Vision: Graphcore](https://mlsanity.com/paper/2301.12082/) (2023)
- [PyramidFlow: High-Resolution Defect Contrastive Localization using Pyramid Normalizing Flow](https://mlsanity.com/paper/2303.02595/) (2023)

## Built on by (most cited first, 30 of 171)

- [AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for Zero-Shot Anomaly Detection](https://mlsanity.com/paper/2407.15795/) (2024)
- [SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model](https://mlsanity.com/paper/2412.04292/) (2024)
- [Harnessing Large Language Models for Training-free Video Anomaly Detection](https://mlsanity.com/paper/2404.01014/) (2024)
- [Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLM](https://mlsanity.com/paper/2406.12235/) (2024)
- [Follow the Rules: Reasoning for Video Anomaly Detection with Large Language Models](https://mlsanity.com/paper/2407.10299/) (2024)
- [Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2502.07601/) (2025)
- [FiLo: Zero-Shot Anomaly Detection by Fine-Grained Description and High-Quality Localization](https://mlsanity.com/paper/2404.13671/) (2024)
- [VCP-CLIP: A visual context prompting model for zero-shot anomaly segmentation](https://mlsanity.com/paper/2407.12276/) (2024)
- [Advancing Video Anomaly Detection: A Concise Review and a New Dataset](https://mlsanity.com/paper/2402.04857/) (2024)
- [Exploring Intrinsic Normal Prototypes within a Single Image for Universal Anomaly Detection](https://mlsanity.com/paper/2503.02424/) (2025)
- [Position: What Can Large Language Models Tell Us about Time Series Analysis](https://mlsanity.com/paper/2402.02713/) (2024)
- [Towards Generic Anomaly Detection and Understanding: Large-scale Visual-linguistic Model (GPT-4V) Takes the Lead](https://mlsanity.com/paper/2311.02782/) (2023)
- [Myriad: Large Multimodal Model by Applying Vision Experts for Industrial Anomaly Detection](https://mlsanity.com/paper/2310.19070/) (2023)
- [Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey](https://mlsanity.com/paper/2407.21794/) (2024)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [The MVTec AD 2 Dataset: Advanced Scenarios for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2503.21622/) (2025)
- [Effectiveness Assessment of Recent Large Vision-Language Models](https://mlsanity.com/paper/2403.04306/) (2024)
- [GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosis](https://mlsanity.com/paper/2411.16778/) (2024)
- [UniVAD: A Training-free Unified Model for Few-shot Visual Anomaly Detection](https://mlsanity.com/paper/2412.03342/) (2024)
- [AnomalyR1: A GRPO-based End-to-end MLLM for Industrial Anomaly Detection](https://mlsanity.com/paper/2504.11914/) (2025)
- [Towards Training-free Anomaly Detection with Vision and Language Foundation Models](https://mlsanity.com/paper/2503.18325/) (2025)
- [GPT-4V-AD: Exploring Grounding Potential of VQA-oriented GPT-4V for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2311.02612/) (2023)
- [Unseen Visual Anomaly Generation](https://mlsanity.com/paper/2406.01078/) (2024)
- [Fine-grained Abnormality Prompt Learning for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2410.10289/) (2024)
- [Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection](https://mlsanity.com/paper/2503.14853/) (2025)
- [PixOOD: Pixel-Level Out-of-Distribution Detection](https://mlsanity.com/paper/2405.19882/) (2024)
- [LogiCode: an LLM-Driven Framework for Logical Anomaly Detection](https://mlsanity.com/paper/2406.04687/) (2024)
- [LLM-3D Print: Large Language Models To Monitor and Control 3D Printing](https://mlsanity.com/paper/2408.14307/) (2024)
- [Learning to Detect Multi-class Anomalies with Just One Normal Image Prompt](https://mlsanity.com/paper/2505.09264/) (2025)
- [Long-Tailed Anomaly Detection with Learnable Class Names](https://mlsanity.com/paper/2403.20236/) (2024)

## Abstract (arXiv)

Large Vision-Language Models (LVLMs) such as MiniGPT-4 and LLaVA have demonstrated the capability of understanding images and achieved remarkable performance in various visual tasks. Despite their strong abilities in recognizing common objects due to extensive training datasets, they lack specific domain knowledge and have a weaker understanding of localized details within objects, which hinders their effectiveness in the Industrial Anomaly Detection (IAD) task. On the other hand, most existing IAD methods only provide anomaly scores and necessitate the manual setting of thresholds to distinguish between normal and abnormal samples, which restricts their practical implementation. In this paper, we explore the utilization of LVLM to address the IAD problem and propose AnomalyGPT, a novel IAD approach based on LVLM. We generate training data by simulating anomalous images and producing corresponding textual descriptions for each image. We also employ an image decoder to provide fine-grained semantic and design a prompt learner to fine-tune the LVLM using prompt embeddings. Our AnomalyGPT eliminates the need for manual threshold adjustments, thus directly assesses the presence and locations of anomalies. Additionally, AnomalyGPT supports multi-turn dialogues and exhibits impressive few-shot in-context learning capabilities. With only one normal shot, AnomalyGPT achieves the state-of-the-art performance with an accuracy of 86.1%, an image-level AUC of 94.1%, and a pixel-level AUC of 95.3% on the MVTec-AD dataset. Code is available at https://github.com/CASIA-IVA-Lab/AnomalyGPT.