# AnomalyMoE: Towards a Language-free Generalist Model for Unified Visual Anomaly Detection

arXiv: https://arxiv.org/abs/2508.06203 · HTML: https://mlsanity.com/paper/2508.06203/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts.
- Builds on 35 of its 43 references that are in the map.
- First posted on arXiv 8 Aug 2025.

## Placed in

- Mixture-of-experts / [Vision mixture-of-experts](https://mlsanity.com/moe/vision-mixture-of-experts/)

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [PaDiM: a Patch Distribution Modeling Framework for Anomaly Detection and Localization](https://mlsanity.com/paper/2011.08785/) (2020)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2207.14315/) (2022)
- [A Unified Model for Multi-class Anomaly Detection](https://mlsanity.com/paper/2206.03687/) (2022)
- [CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information](https://mlsanity.com/paper/2006.12013/) (2020)
- [The MVTec 3D-AD Dataset for Unsupervised 3D Anomaly Detection and Localization](https://mlsanity.com/paper/2112.09045/) (2021)
- [FLatten Transformer: Vision Transformer using Focused Linear Attention](https://mlsanity.com/paper/2308.00442/) (2023)
- [AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models](https://mlsanity.com/paper/2308.15366/) (2023)
- [Dinomaly: The Less Is More Philosophy in Multi-Class Unsupervised Anomaly Detection](https://mlsanity.com/paper/2405.14325/) (2024)
- [BMAD: Benchmarks for Medical Anomaly Detection](https://mlsanity.com/paper/2306.11876/) (2023)
- [Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images](https://mlsanity.com/paper/2403.12570/) (2024)
- [ReContrast: Domain-Specific Anomaly Detection via Contrastive Reconstruction](https://mlsanity.com/paper/2306.02602/) (2023)
- [Follow the Rules: Reasoning for Video Anomaly Detection with Large Language Models](https://mlsanity.com/paper/2407.10299/) (2024)
- [FiLo: Zero-Shot Anomaly Detection by Fine-Grained Description and High-Quality Localization](https://mlsanity.com/paper/2404.13671/) (2024)
- [PO3AD: Predicting Point Offsets toward Better 3D Point Cloud Anomaly Detection](https://mlsanity.com/paper/2412.12617/) (2024)
- [Look Inside for More: Internal Spatial Modality Perception for 3D Anomaly Detection](https://mlsanity.com/paper/2412.13461/) (2024)
- [PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection](https://mlsanity.com/paper/2410.00320/) (2024)
- [UniVAD: A Training-free Unified Model for Few-shot Visual Anomaly Detection](https://mlsanity.com/paper/2412.03342/) (2024)
- [Towards Training-free Anomaly Detection with Vision and Language Foundation Models](https://mlsanity.com/paper/2503.18325/) (2025)
- [Component-aware anomaly detection framework for adjustable and logical industrial visual inspection](https://mlsanity.com/paper/2305.08509/) (2023)
- [Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning](https://mlsanity.com/paper/2405.16869/) (2024)
- [CSAD: Unsupervised Component Segmentation for Logical Anomaly Detection](https://mlsanity.com/paper/2408.15628/) (2024)

## Built on by (most cited first, 8 of 8)

- [Towards One-for-All Anomaly Detection for Tabular Data](https://mlsanity.com/paper/2603.14407/) (2026)
- [Resolving Primitive-Sharing Ambiguity in Long-Tailed TLS-Based Industrial MEP Point Cloud Segmentation via Spatial Context Constraints](https://mlsanity.com/paper/2601.19128/) (2026)
- [IMMoE: Incomplete Multi-View Anomaly Detection via Mixture of View Experts Fusion](https://mlsanity.com/paper/2607.19032/) (2026)
- [StructCore: Structure-Aware Image-Level Scoring for Training-Free Unsupervised Anomaly Detection](https://mlsanity.com/paper/2602.17048/) (2026)
- [MoECLIP: Patch-Specialized Experts for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2603.03101/) (2026)
- [LogiCo: A Unified Framework for Logical and Structural Anomaly Detection](https://mlsanity.com/paper/2606.28688/) (2026)
- [UniVAD v2: Unified Visual Anomaly Detection via Support-Conditioned Boundary Construction](https://mlsanity.com/paper/2606.29714/) (2026)
- [When Point Clouds Outperform Pixels: Rethinking Zero-Shot Multimodal Anomaly Detection](https://mlsanity.com/paper/2609.25793/) (2026)

## Abstract (arXiv)

Anomaly detection is a critical task across numerous domains and modalities, yet existing methods are often highly specialized, limiting their generalizability. These specialized models, tailored for specific anomaly types like textural defects or logical errors, typically exhibit limited performance when deployed outside their designated contexts. To overcome this limitation, we propose AnomalyMoE, a novel and universal anomaly detection framework based on a Mixture-of-Experts (MoE) architecture. Our key insight is to decompose the complex anomaly detection problem into three distinct semantic hierarchies: local structural anomalies, component-level semantic anomalies, and global logical anomalies. AnomalyMoE correspondingly employs three dedicated expert networks at the patch, component, and global levels, and is specialized in reconstructing features and identifying deviations at its designated semantic level. This hierarchical design allows a single model to concurrently understand and detect a wide spectrum of anomalies. Furthermore, we introduce an Expert Information Repulsion (EIR) module to promote expert diversity and an Expert Selection Balancing (ESB) module to ensure the comprehensive utilization of all experts. Experiments on 8 challenging datasets spanning industrial imaging, 3D point clouds, medical imaging, video surveillance, and logical anomaly detection demonstrate that AnomalyMoE establishes new state-of-the-art performance, significantly outperforming specialized methods in their respective domains.