# Towards Training-free Anomaly Detection with Vision and Language Foundation Models

arXiv: https://arxiv.org/abs/2503.18325 · HTML: https://mlsanity.com/paper/2503.18325/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 49 references that are in the map.
- First posted on arXiv 24 Mar 2025.

## Builds on (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Towards Total Recall in Industrial Anomaly Detection](https://mlsanity.com/paper/2106.08265/) (2021)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [PaDiM: a Patch Distribution Modeling Framework for Anomaly Detection and Localization](https://mlsanity.com/paper/2011.08785/) (2020)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2207.14315/) (2022)
- [Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models](https://mlsanity.com/paper/2403.18814/) (2024)
- [WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation](https://mlsanity.com/paper/2303.14814/) (2023)
- [Sub-Image Anomaly Detection with Deep Pyramid Correspondences](https://mlsanity.com/paper/2005.02357/) (2020)
- [AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2310.18961/) (2023)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [A Unified Model for Multi-class Anomaly Detection](https://mlsanity.com/paper/2206.03687/) (2022)
- [SimpleNet: A Simple Network for Image Anomaly Detection and Localization](https://mlsanity.com/paper/2303.15140/) (2023)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models](https://mlsanity.com/paper/2308.15366/) (2023)
- [Compositional Chain-of-Thought Prompting for Large Multimodal Models](https://mlsanity.com/paper/2311.17076/) (2023)
- [EfficientAD: Accurate Visual Anomaly Detection at Millisecond-Level Latencies](https://mlsanity.com/paper/2303.14535/) (2023)
- [SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding](https://mlsanity.com/paper/2310.15308/) (2023)
- [PromptAD: Learning Prompts with only Normal Samples for Few-Shot Anomaly Detection](https://mlsanity.com/paper/2404.05231/) (2024)

## Built on by (most cited first, 26 of 26)

- [LAD-Reasoner: Tiny Multimodal Models are Good Reasoners for Logical Anomaly Detection](https://mlsanity.com/paper/2504.12749/) (2025)
- [AnomalyMoE: Towards a Language-free Generalist Model for Unified Visual Anomaly Detection](https://mlsanity.com/paper/2508.06203/) (2025)
- [Foundation Visual Encoders Are Secretly Few-Shot Anomaly Detectors](https://mlsanity.com/paper/2510.01934/) (2025)
- [A Comprehensive Survey for Real-World Industrial Defect Detection: Challenges, Approaches, and Prospects](https://mlsanity.com/paper/2507.13378/) (2025)
- [From Benchmarks to Reality: Advancing Visual Anomaly Detection by the VAND 3.0 Challenge](https://mlsanity.com/paper/2509.17615/) (2025)
- [Multi-turn Physics-informed Vision-language Model for Physics-grounded Anomaly Detection](https://mlsanity.com/paper/2603.15237/) (2026)
- [Time-reversed Flow Matching with Worst Transport in High-dimensional Latent Space for Image Anomaly Detection](https://mlsanity.com/paper/2508.05461/) (2025)
- [Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindness](https://mlsanity.com/paper/2602.19615/) (2026)
- [PASTA: Vision Transformer Patch Aggregation for Weakly Supervised Target and Anomaly Segmentation](https://mlsanity.com/paper/2604.09701/) (2026)
- [AnomalyAgent: Training-Free Agentic Models for Zero-/Few-Shot Anomaly Detection](https://mlsanity.com/paper/2605.30140/) (2026)
- [UniSLAD: A Unified Framework for Structural and Logical Industrial Visual Anomaly Detection](https://mlsanity.com/paper/2606.20768/) (2026)
- [A VLM Answer Is Not an Anomaly Score: Rank Compression Across Image and Video Anomaly Detection](https://mlsanity.com/paper/2608.21244/) (2026)
- [TMUAD: Enhancing Logical Capabilities in Unified Anomaly Detection Models with a Text Memory Bank](https://mlsanity.com/paper/2508.21795/) (2025)
- [Language-in-the-Loop Culvert Inspection on the Erie Canal](https://mlsanity.com/paper/2509.21370/) (2025)
- [Adversarial Robustness in Zero-Shot Learning:An Empirical Study on Class and Concept-Level Vulnerabilities](https://mlsanity.com/paper/2512.18651/) (2025)
- [Interpretable Logical Anomaly Classification via Constraint Decomposition and Instruction Fine-Tuning](https://mlsanity.com/paper/2602.03530/) (2026)
- [BUSSARD: Normalizing Flows for Bijective Universal Scene-Specific Anomalous Relationship Detection](https://mlsanity.com/paper/2603.16645/) (2026)
- [Fine-Tuning Vision-Language Models for Understanding Current Damage and Scoring Priority with Quality Guard Agent](https://mlsanity.com/paper/2605.27452/) (2026)
- [A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision](https://mlsanity.com/paper/2606.01992/) (2026)
- [LADBench: A Benchmark for Logical Fault Detection in Images](https://mlsanity.com/paper/2606.17433/) (2026)
- [MATCH: Flow Matching for Multi-View Anomaly Detection](https://mlsanity.com/paper/2606.24375/) (2026)
- [LogiCo: A Unified Framework for Logical and Structural Anomaly Detection](https://mlsanity.com/paper/2606.28688/) (2026)
- [DuoAD: Leveraging [CLS] Dual Characteristics for Training-Free Few-Shot Anomaly Detection](https://mlsanity.com/paper/2607.23924/) (2026)
- [Multi-Modal Anomaly Detection: A Survey](https://mlsanity.com/paper/2608.24937/) (2026)
- [Robust Logical Anomaly Detection under Capture Shifts via Synthetic Anomalies](https://mlsanity.com/paper/2610.18295/) (2026)
- [CORE: Contrast-and-Reflect for Self-Supervised Prompt Optimization in Anomaly Synthesis](https://mlsanity.com/paper/2610.19610/) (2026)

## Abstract (arXiv)

Anomaly detection is valuable for real-world applications, such as industrial quality inspection. However, most approaches focus on detecting local structural anomalies while neglecting compositional anomalies incorporating logical constraints. In this paper, we introduce LogSAD, a novel multi-modal framework that requires no training for both Logical and Structural Anomaly Detection. First, we propose a match-of-thought architecture that employs advanced large multi-modal models (i.e. GPT-4V) to generate matching proposals, formulating interests and compositional rules of thought for anomaly detection. Second, we elaborate on multi-granularity anomaly detection, consisting of patch tokens, sets of interests, and composition matching with vision and language foundation models. Subsequently, we present a calibration module to align anomaly scores from different detectors, followed by integration strategies for the final decision. Consequently, our approach addresses both logical and structural anomaly detection within a unified framework and achieves state-of-the-art results without the need for training, even when compared to supervised approaches, highlighting its robustness and effectiveness. Code is available at https://github.com/zhang0jhon/LogSAD.