# Segment Any Anomaly without Training via Hybrid Prompt Regularization

arXiv: https://arxiv.org/abs/2305.10724 · HTML: https://mlsanity.com/paper/2305.10724/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 107 references that are in the map.
- First posted on arXiv 18 May 2023.

## Builds on (most cited first, 30 of 57)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Towards Total Recall in Industrial Anomaly Detection](https://mlsanity.com/paper/2106.08265/) (2021)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [DRAEM -- A discriminatively trained reconstruction embedding for surface anomaly detection](https://mlsanity.com/paper/2108.07610/) (2021)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [Anomaly Detection via Reverse Distillation from One-Class Embedding](https://mlsanity.com/paper/2201.10703/) (2022)
- [Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks](https://mlsanity.com/paper/2206.08916/) (2022)
- [SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2207.14315/) (2022)
- [Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddings](https://mlsanity.com/paper/1911.02357/) (2019)
- [WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation](https://mlsanity.com/paper/2303.14814/) (2023)
- [Prompting Visual-Language Models for Efficient Video Understanding](https://mlsanity.com/paper/2112.04478/) (2021)
- [Multiresolution Knowledge Distillation for Anomaly Detection](https://mlsanity.com/paper/2011.11108/) (2020)
- [Exploring Visual Prompts for Adapting Large-Scale Models](https://mlsanity.com/paper/2203.17274/) (2022)
- [What does CLIP know about a red circle? Visual prompt engineering for VLMs](https://mlsanity.com/paper/2304.06712/) (2023)
- [Patch SVDD: Patch-level SVDD for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2006.16067/) (2020)
- [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023)

## Built on by (most cited first, 30 of 60)

- [AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for Zero-Shot Anomaly Detection](https://mlsanity.com/paper/2407.15795/) (2024)
- [AnomalyDiffusion: Few-Shot Anomaly Image Generation with Diffusion Model](https://mlsanity.com/paper/2312.05767/) (2023)
- [PromptAD: Learning Prompts with only Normal Samples for Few-Shot Anomaly Detection](https://mlsanity.com/paper/2404.05231/) (2024)
- [Real3D-AD: A Dataset of Point Cloud Anomaly Detection](https://mlsanity.com/paper/2309.13226/) (2023)
- [Review of Large Vision Models and Visual Prompt Engineering](https://mlsanity.com/paper/2307.00855/) (2023)
- [AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIP](https://mlsanity.com/paper/2503.06661/) (2025)
- [A Survey on Visual Anomaly Detection: Challenge, Approach, and Prospect](https://mlsanity.com/paper/2401.16402/) (2024)
- [Towards Scalable 3D Anomaly Detection and Localization: A Benchmark via 3D Anomaly Synthesis and A Self-Supervised Learning Network](https://mlsanity.com/paper/2311.14897/) (2023)
- [CLIP-AD: A Language-Guided Staged Dual-Path Model for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2311.00453/) (2023)
- [MuSc: Zero-Shot Industrial Anomaly Classification and Segmentation with Mutual Scoring of the Unlabeled Images](https://mlsanity.com/paper/2401.16753/) (2024)
- [FiLo: Zero-Shot Anomaly Detection by Fine-Grained Description and High-Quality Localization](https://mlsanity.com/paper/2404.13671/) (2024)
- [VCP-CLIP: A visual context prompting model for zero-shot anomaly segmentation](https://mlsanity.com/paper/2407.12276/) (2024)
- [ClipSAM: CLIP and SAM Collaboration for Zero-Shot Anomaly Segmentation](https://mlsanity.com/paper/2401.12665/) (2024)
- [Exploring Plain ViT Reconstruction for Multi-class Unsupervised Anomaly Detection](https://mlsanity.com/paper/2312.07495/) (2023)
- [Towards Generic Anomaly Detection and Understanding: Large-scale Visual-linguistic Model (GPT-4V) Takes the Lead](https://mlsanity.com/paper/2311.02782/) (2023)
- [DiAD: A Diffusion-based Framework for Multi-class Anomaly Detection](https://mlsanity.com/paper/2312.06607/) (2023)
- [GPT-4V-AD: Exploring Grounding Potential of VQA-oriented GPT-4V for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2311.02612/) (2023)
- [Target before Shooting: Accurate Anomaly Detection and Localization under One Millisecond via Cascade Patch Retrieval](https://mlsanity.com/paper/2308.06748/) (2023)
- [AnomalyXFusion: Multi-modal Anomaly Synthesis with Diffusion](https://mlsanity.com/paper/2404.19444/) (2024)
- [Spiking-PhysFormer: Camera-Based Remote Photoplethysmography with Parallel Spike-driven Transformer](https://mlsanity.com/paper/2402.04798/) (2024)
- [Learning to Detect Multi-class Anomalies with Just One Normal Image Prompt](https://mlsanity.com/paper/2505.09264/) (2025)
- [OmniAD: Detect and Understand Industrial Anomaly via Multimodal Reasoning](https://mlsanity.com/paper/2505.22039/) (2025)
- [RepViT-SAM: Towards Real-Time Segmenting Anything](https://mlsanity.com/paper/2312.05760/) (2023)
- [Boosting Global-Local Feature Matching via Anomaly Synthesis for Multi-Class Point Cloud Anomaly Detection](https://mlsanity.com/paper/2505.07375/) (2025)
- [UGainS: Uncertainty Guided Anomaly Instance Segmentation](https://mlsanity.com/paper/2308.02046/) (2023)
- [Towards Zero-shot Point Cloud Anomaly Detection: A Multi-View Projection Framework](https://mlsanity.com/paper/2409.13162/) (2024)
- [Long-Tailed Anomaly Detection with Learnable Class Names](https://mlsanity.com/paper/2403.20236/) (2024)
- [Optimizing PatchCore for Few/many-shot Anomaly Detection](https://mlsanity.com/paper/2307.10792/) (2023)
- [RAD: A Comprehensive Dataset for Benchmarking the Robustness of Image Anomaly Detection](https://mlsanity.com/paper/2406.07176/) (2024)
- [UrbanSAM: Learning Invariance-Inspired Adapters for Segment Anything Models in Urban Construction](https://mlsanity.com/paper/2502.15199/) (2025)

## Abstract (arXiv)

We present a novel framework, i.e., Segment Any Anomaly + (SAA+), for zero-shot anomaly segmentation with hybrid prompt regularization to improve the adaptability of modern foundation models. Existing anomaly segmentation models typically rely on domain-specific fine-tuning, limiting their generalization across countless anomaly patterns. In this work, inspired by the great zero-shot generalization ability of foundation models like Segment Anything, we first explore their assembly to leverage diverse multi-modal prior knowledge for anomaly localization. For non-parameter foundation model adaptation to anomaly segmentation, we further introduce hybrid prompts derived from domain expert knowledge and target image context as regularization. Our proposed SAA+ model achieves state-of-the-art performance on several anomaly segmentation benchmarks, including VisA, MVTec-AD, MTD, and KSDD2, in the zero-shot setting. We will release the code at hrefhttps://github.com/caoyunkang/Segment-Any-Anomalyhttps://github.com/caoyunkang/Segment-Any-Anomaly.