# AnomalyAgent: Training-Free Agentic Models for Zero-/Few-Shot Anomaly Detection

arXiv: https://arxiv.org/abs/2605.30140 · HTML: https://mlsanity.com/paper/2605.30140/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 34 of its 38 references that are in the map.
- First posted on arXiv 28 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)
- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data](https://mlsanity.com/paper/2107.10833/) (2021)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [Towards Total Recall in Industrial Anomaly Detection](https://mlsanity.com/paper/2106.08265/) (2021)
- [WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation](https://mlsanity.com/paper/2303.14814/) (2023)
- [Multiresolution Knowledge Distillation for Anomaly Detection](https://mlsanity.com/paper/2011.11108/) (2020)
- [AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2310.18961/) (2023)
- [AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models](https://mlsanity.com/paper/2308.15366/) (2023)
- [APRIL-GAN: A Zero-/Few-Shot Anomaly Classification and Segmentation Method for CVPR 2023 VAND Workshop Challenge Tracks 1&2: 1st Place on Zero-shot AD and 4th Place on Few-shot AD](https://mlsanity.com/paper/2305.17382/) (2023)
- [AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for Zero-Shot Anomaly Detection](https://mlsanity.com/paper/2407.15795/) (2024)
- [PromptAD: Learning Prompts with only Normal Samples for Few-Shot Anomaly Detection](https://mlsanity.com/paper/2404.05231/) (2024)
- [Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Prompts](https://mlsanity.com/paper/2403.06495/) (2024)
- [Attention Based Glaucoma Detection: A Large-scale Database and CNN Model](https://mlsanity.com/paper/1903.10831/) (2019)
- [AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIP](https://mlsanity.com/paper/2503.06661/) (2025)
- [A Survey on Visual Anomaly Detection: Challenge, Approach, and Prospect](https://mlsanity.com/paper/2401.16402/) (2024)
- [Generative and Contrastive Self-Supervised Learning for Graph Anomaly Detection](https://mlsanity.com/paper/2108.09896/) (2021)
- [Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2502.07601/) (2025)
- [FiLo: Zero-Shot Anomaly Detection by Fine-Grained Description and High-Quality Localization](https://mlsanity.com/paper/2404.13671/) (2024)
- [MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection](https://mlsanity.com/paper/2410.09453/) (2024)
- [AnomalyR1: A GRPO-based End-to-end MLLM for Industrial Anomaly Detection](https://mlsanity.com/paper/2504.11914/) (2025)
- [Deep Learning for Video Anomaly Detection: A Review](https://mlsanity.com/paper/2409.05383/) (2024)
- [Towards Training-free Anomaly Detection with Vision and Language Foundation Models](https://mlsanity.com/paper/2503.18325/) (2025)
- [ResAD: A Simple Framework for Class Generalizable Anomaly Detection](https://mlsanity.com/paper/2410.20047/) (2024)
- [Fine-grained Abnormality Prompt Learning for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2410.10289/) (2024)
- [DictAS: A Framework for Class-Generalizable Few-Shot Anomaly Segmentation via Dictionary Lookup](https://mlsanity.com/paper/2508.13560/) (2025)

## Built on by (most cited first, 1 of 1)

- [Copy What Is Seen, Generate What Is Not: Training-Free Anomaly-Aware Video Restoration](https://mlsanity.com/paper/2609.18836/) (2026)

## Abstract (arXiv)

Benefiting from generalizability of vision-language models (VLMs) such as CLIP, many zero-/few-shot anomaly detection (AD) approaches have achieved impressive detection performance across various datasets. Nevertheless, they require substantial training on large auxiliary datasets to adapt VLMs to anomaly detection, and their inference largely relies on visual-text embedding similarity-based anomaly scores, lacking reasoning abilities to detect complex anomalies that require in-depth contextual understanding. To address this limitation, we propose AnomalyAgent, a novel training-free, agentic framework that leverages the advanced reasoning and generalization capabilities of multimodal large language models (MLLMs) for anomaly detection. The key ingredients include 1) a comprehensive anomaly-centric toolset that enables adaptive MLLM-driven, agentic anomaly reasoning in zero-shot settings, and 2) a customized memory module that grounds anomaly reasoning with few-shot, in-context reference examples. We extend evaluation beyond the detection of simple anomalies (e.g., surface defects like cracks and dents and clear lesions) in widely used benchmarks to more diverse types of anomalies such as logical/contextual anomalies in logistics and manufacturing settings. Extensive experiment results demonstrate that our AnomalyAgent achieves substantially better performance compared to training-free VLM-based AD and generic agentic methods, highlighting its superior generalization capability in both zero-shot and few-shot anomaly detection settings. The code implementation can be find at this address.