# AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimization

arXiv: https://arxiv.org/abs/2508.04175 · HTML: https://mlsanity.com/paper/2508.04175/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning.
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Placed in

- LLM reasoning / [Chain-of-thought reasoning for driving](https://mlsanity.com/lr/chain-of-thought-reasoning-for-driving/)
- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 17 of 17)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Towards Total Recall in Industrial Anomaly Detection](https://mlsanity.com/paper/2106.08265/) (2021)
- [Visual-RFT: Visual Reinforcement Fine-Tuning](https://mlsanity.com/paper/2503.01785/) (2025)
- [Anomaly Detection via Reverse Distillation from One-Class Embedding](https://mlsanity.com/paper/2201.10703/) (2022)
- [GRIT: Teaching MLLMs to Think with Images](https://mlsanity.com/paper/2505.15879/) (2025)
- [RealNet: A Feature Selection Network with Realistic Synthetic Anomaly for Anomaly Detection](https://mlsanity.com/paper/2403.05897/) (2024)
- [PyramidFlow: High-Resolution Defect Contrastive Localization using Pyramid Normalizing Flow](https://mlsanity.com/paper/2303.02595/) (2023)
- [Towards Generic Anomaly Detection and Understanding: Large-scale Visual-linguistic Model (GPT-4V) Takes the Lead](https://mlsanity.com/paper/2311.02782/) (2023)
- [Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach](https://mlsanity.com/paper/2503.21819/) (2025)
- [OmniAD: Detect and Understand Industrial Anomaly via Multimodal Reasoning](https://mlsanity.com/paper/2505.22039/) (2025)
- [A Comprehensive Library for Benchmarking Multi-class Visual Anomaly Detection](https://mlsanity.com/paper/2406.03262/) (2024)

## Built on by (most cited first, 8 of 8)

- [Towards Explainable Industrial Anomaly Detection via Knowledge-Guided Latent Reasoning](https://mlsanity.com/paper/2602.09850/) (2026)
- [AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparison](https://mlsanity.com/paper/2603.13779/) (2026)
- [AgentIAD: Agentic Industrial Anomaly Detection via Adaptive Memory Augmentation](https://mlsanity.com/paper/2512.13671/) (2025)
- [ABounD: Adversarial Boundary-Driven Few-Shot Learning for Multi-Class Anomaly Detection](https://mlsanity.com/paper/2511.22436/) (2025)
- [VD-DeepStack: Bridging Visual Comparison and Language Reasoning for Few-Shot Anomaly Detection](https://mlsanity.com/paper/2609.34949/) (2026)
- [Global Logic and Local Search: Dual-Stream Multimodal In-Context Learning for Verifiable Industrial Anomaly Detection](https://mlsanity.com/paper/2607.03817/) (2026)
- [Industrial Anomaly Detection via Defect-Grounded Reasoning in Visual Latent Space](https://mlsanity.com/paper/2609.29457/) (2026)
- [TED:Text-Axis Evidence Decomposition for Prompted Anomaly Localization](https://mlsanity.com/paper/2609.39033/) (2026)

## Abstract (arXiv)

While Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities across diverse domains, their application to specialized anomaly detection (AD) remains constrained by domain adaptation challenges. Existing Group Relative Policy Optimization (GRPO) based approaches suffer from two critical limitations: inadequate training data utilization when models produce uniform responses, and insufficient supervision over reasoning processes that encourage immediate binary decisions without deliberative analysis. We propose a comprehensive framework addressing these limitations through two synergistic innovations. First, we introduce a multi-stage deliberative reasoning process that guides models from region identification to focused examination, generating diverse response patterns essential for GRPO optimization while enabling structured supervision over analytical workflows. Second, we develop a fine-grained reward mechanism incorporating classification accuracy and localization supervision, transforming binary feedback into continuous signals that distinguish genuine analytical insight from spurious correctness. Comprehensive evaluation across multiple industrial datasets demonstrates substantial performance improvements in adapting general vision-language models to specialized anomaly detection. Our method achieves superior accuracy with efficient adaptation of existing annotations, effectively bridging the gap between general-purpose MLLM capabilities and the fine-grained visual discrimination required for detecting subtle manufacturing defects and structural irregularities.