mlsanityData to 7 Oct 2026

Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models

arXiv 2502.07601 ↗First posted 11 Feb 2025cs.CV cs.CLcited by 54

On the map: LLM reasoning → Chain-of-thought reasoning for driving

also in 1 more
Abstract (arXiv)

Zero-Shot Anomaly Detection (ZSAD) is an emerging AD paradigm. Unlike the traditional unsupervised AD setting that requires a large number of normal samples to train a model, ZSAD is more practical for handling data-restricted real-world scenarios. Recently, Multimodal Large Language Models (MLLMs) have shown revolutionary reasoning capabilities in various vision tasks. However, the reasoning of image abnormalities remains underexplored due to the lack of corresponding datasets and benchmarks. To facilitate research in AD & reasoning, we establish the first visual instruction tuning dataset, Anomaly-Instruct-125k, and the evaluation benchmark, VisA-D&R. Through investigation with our benchmark, we reveal that current MLLMs like GPT-4o cannot accurately detect and describe fine-grained anomalous details in images. To address this, we propose Anomaly-OneVision (Anomaly-OV), the first specialist visual assistant for ZSAD and reasoning. Inspired by human behavior in visual inspection, Anomaly-OV leverages a Look-Twice Feature Matching (LTFM) mechanism to adaptively select and emphasize abnormal visual tokens. Extensive experiments demonstrate that Anomaly-OV achieves significant improvements over advanced generalist models in both detection and reasoning. Extensions to medical and 3D AD are provided for future study. The link to our project page: https://xujiacong.github.io/Anomaly-OV/

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 90 of its 112 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

90 of 112 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Follow the Rules: Reasoning for Video Anomaly Detection with Large Language Models14 Jul
  2. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving6 Dec

Other references · 88

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  7. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  10. 2021Towards Total Recall in Industrial Anomaly Detectioncited by 772
  11. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  12. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  13. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  14. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  15. 2021CutPaste: Self-Supervised Learning for Anomaly Detection and Localizationcited by 529
  16. 2021DRAEM -- A discriminatively trained reconstruction embedding for surface anomaly detectioncited by 510
  17. 2022Anomaly Detection via Reverse Distillation from One-Class Embeddingcited by 419
  18. 2022SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentationcited by 397
  19. 2019Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddingscited by 386
  20. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 311
  21. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  22. 2023AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detectioncited by 270
  23. 2022A Unified Model for Multi-class Anomaly Detectioncited by 267
  24. 2022Diffusion Models for Medical Anomaly Detectioncited by 231
  25. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  26. 2021VT-ADL: A Vision Transformer Network for Image Anomaly Detection and Localizationcited by 222
  27. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  28. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  29. 2021The MVTec 3D-AD Dataset for Unsupervised 3D Anomaly Detection and Localizationcited by 174
  30. 2023AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Modelscited by 169
  31. 2022DeSTSeg: Segmentation Guided Denoising Student-Teacher for Anomaly Detectioncited by 153
  32. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  33. 2020Deep Learning for Medical Anomaly Detection -- A Surveycited by 127
  34. 2024AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for Zero-Shot Anomaly Detectioncited by 126
  35. 2022CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentationcited by 118
  36. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 117
  37. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 116
  38. 2024RealNet: A Feature Selection Network with Realistic Synthetic Anomaly for Anomaly Detectioncited by 116

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 31.

54 papers.

Sort:

By area:

  1. 2025AnomalyR1: A GRPO-based End-to-end MLLM for Industrial Anomaly Detectioncited by 28
  2. 2026Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysiscited by 15
  3. 2025Taming Anomalies with Down-Up Sampling Networks: Group Center Preserving Reconstruction for 3D Anomaly Detectioncited by 11
  4. 2025LAD-Reasoner: Tiny Multimodal Models are Good Reasoners for Logical Anomaly Detectioncited by 9
  5. 2026AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparisoncited by 5
  6. 2025SAGE: A Visual Language Model for Anomaly Detection via Fact Enhancement and Entropy-aware Alignmentcited by 4
  7. 2025DP-DGAD: A Generalist Dynamic Graph Anomaly Detector with Dynamic Prototypescited by 4
  8. 2026JUDO: A Juxtaposed Domain-Oriented Multimodal Reasoner for Industrial Anomaly QAcited by 4
  9. 2025GenCLIP: Generalizing CLIP Prompts for Zero-shot Anomaly Detectioncited by 3
  10. 2025Multimodal RAG-driven Anomaly Detection and Classification in Laser Powder Bed Fusion using Large Language Modelscited by 3
  11. 2025SketchMind: A Multi-Agent Cognitive Framework for Assessing Student-Drawn Scientific Sketchescited by 3
  12. 2025VLMDiff: Leveraging Vision-Language Models for Multi-Class Anomaly Detection with Diffusioncited by 3
38 more
  1. 2025A Comprehensive Survey for Real-World Industrial Defect Detection: Challenges, Approaches, and Prospectscited by 2
  2. 2025Q-Former Autoencoder: A Modern Framework for Medical Anomaly Detectioncited by 2
  3. 2026EAGLE: Expert-Augmented Attention Guidance for Tuning-Free Industrial Anomaly Detection in Multimodal Large Language Modelscited by 2
  4. 2026Back to Point: Exploring Point-Language Models for Zero-Shot 3D Anomaly Detectioncited by 2
  5. 2025IQE-CLIP: Instance-aware Query Embedding for Zero-/Few-shot Anomaly Detection in Medical Domaincited by 1
  6. 2025CoPS: Conditional Prompt Synthesis for Zero-Shot Anomaly Detectioncited by 1
  7. 2025IADGPT: Unified LVLM for Few-Shot Industrial Anomaly Detection, Localization, and Reasoning via In-Context Learningcited by 1
  8. 2025A Contrastive Learning-Guided Confident Meta-learning for Zero Shot Anomaly Detectioncited by 1
  9. 2025Efficient Odd-One-Out Anomaly Detectioncited by 1
  10. 2025Action Hints: Semantic Typicality and Context Uniqueness for Generalizable Skeleton-based Video Anomaly Detectioncited by 1
  11. 2025ABounD: Adversarial Boundary-Driven Few-Shot Learning for Multi-Class Anomaly Detectioncited by 1
  12. 2025Silhouette-based Gait Foundation Modelcited by 1
  13. 2026M3-AD: Reflection-aware Multi-modal, Multi-category, and Multi-dimensional Benchmark and Framework for Industrial Anomaly Detectioncited by 1
  14. 2026AG-VAS: Anchor-Guided Zero-Shot Visual Anomaly Segmentation with Large Multimodal Modelscited by 1
  15. 2026Topo-R1: Detecting Topological Anomalies via Vision-Language Modelscited by 1
  16. 2026MMR-AD: A Large-Scale Multimodal Dataset for Benchmarking General Anomaly Detection with Multimodal Large Language Modelscited by 1
  17. 2026Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledgecited by 1
  18. 2026AnomalyAgent: Training-Free Agentic Models for Zero-/Few-Shot Anomaly Detectioncited by 1
  19. 2026CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detectioncited by 1
  20. 2026OPD-IAD: From Language Judgment to Industrial Anomaly Detection via On-Policy Self-Distillationcited by 1
  21. 2026A VLM Answer Is Not an Anomaly Score: Rank Compression Across Image and Video Anomaly Detectioncited by 1
  22. 2025ADSeeker: A Knowledge-Grounded Reasoning Framework for Industry Anomaly Detection and Reasoningcited by 0
  23. 2025TMUAD: Enhancing Logical Capabilities in Unified Anomaly Detection Models with a Text Memory Bankcited by 0
  24. 2025Noise & pattern: identity-anchored Tikhonov regularization for robust structural anomaly detectioncited by 0
  25. 2025Defect-aware Hybrid Prompt Optimization via Progressive Tuning for Zero-Shot Multi-type Anomaly Detection and Segmentationcited by 0
  26. 2026SSVP: Synergistic Semantic-Visual Prompting for Industrial Zero-Shot Anomaly Detectioncited by 0
  27. 2026From Prediction to Diagnosis: Reasoning-Aware AI for Photovoltaic Defect Inspectioncited by 0
  28. 2026MEDIC-AD: Towards Medical Vision-Language Model's Clinical Intelligencecited by 0
  29. 2026Latent Anomaly Knowledge Excavation: Unveiling Sparse Sensitive Neurons in Vision-Language Modelscited by 0
  30. 2026Agri-CPJ: A Training-Free Explainable Framework for Agricultural Pest Diagnosis Using Caption-Prompt-Judge and LLM-as-a-Judgecited by 0
  31. 2026IndusAgent: Reinforcing Open-Vocabulary Industrial Anomaly Detection with Agentic Toolscited by 0
  32. 2026Beyond Normal References: Discriminative Few-Shot Anomaly Detectioncited by 0
  33. 2026Fine-Tuning Vision-Language Models for Understanding Current Damage and Scoring Priority with Quality Guard Agentcited by 0
  34. 2026A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decisioncited by 0
  35. 2026Value-order Decomposition for Generalist Anomaly Detectioncited by 0
  36. 2026DifferAD-R1: A Difference-Guided IndustrialAnomaly Localization with Multimodal LargeLanguage Modelscited by 0
  37. 2026GenAU: Language-Grounded Industrial Anomaly Understanding with Vision-Language Modelscited by 0
  38. 2026PADFormer: Pose-agnostic Anomaly Detection from Sparse View Imagescited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer
  2. 2024VMAD: Visual-enhanced Multimodal Large Language Model for Zero-Shot Anomaly Detection
  3. 2026Are Multimodal LLMs Ready for Surveillance? A Reality Check on Zero-Shot Anomaly Detection in the Wild
  4. 2026TIPS Over Tricks: Simple Prompts for Effective Zero-shot Anomaly Detection
  5. 2025PromptMoE: Generalizable Zero-Shot Anomaly Detection via Visually-Guided Prompt Mixtures
  6. 2023CLIP-AD: A Language-Guided Staged Dual-Path Model for Zero-shot Anomaly Detection
  7. 2025KAnoCLIP: Zero-Shot Anomaly Detection through Knowledge-Driven Prompt Learning and Enhanced Cross-Modal Integration
  8. 2025LogicAD: Explainable Anomaly Detection via VLM-based Text Feature Extraction