# Latent Anomaly Knowledge Excavation: Unveiling Sparse Sensitive Neurons in Vision-Language Models

arXiv: https://arxiv.org/abs/2604.07802 · HTML: https://mlsanity.com/paper/2604.07802/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 43 of its 50 references that are in the map.
- First posted on arXiv 9 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Towards Total Recall in Industrial Anomaly Detection](https://mlsanity.com/paper/2106.08265/) (2021)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2207.14315/) (2022)
- [Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddings](https://mlsanity.com/paper/1911.02357/) (2019)
- [WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation](https://mlsanity.com/paper/2303.14814/) (2023)
- [AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2310.18961/) (2023)
- [AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for Zero-Shot Anomaly Detection](https://mlsanity.com/paper/2407.15795/) (2024)
- [RealNet: A Feature Selection Network with Realistic Synthetic Anomaly for Anomaly Detection](https://mlsanity.com/paper/2403.05897/) (2024)
- [AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIP](https://mlsanity.com/paper/2503.06661/) (2025)
- [Sparse autoencoders reveal selective remapping of visual concepts during adaptation](https://mlsanity.com/paper/2412.05276/) (2024)
- [Interpreting CLIP with Hierarchical Sparse Autoencoders](https://mlsanity.com/paper/2502.20578/) (2025)
- [CLIP-AD: A Language-Guided Staged Dual-Path Model for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2311.00453/) (2023)
- [Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2502.07601/) (2025)
- [Interpreting the Second-Order Effects of Neurons in CLIP](https://mlsanity.com/paper/2406.04341/) (2024)
- [Dual-Interrelated Diffusion Model for Few-Shot Anomaly Image Generation](https://mlsanity.com/paper/2408.13509/) (2024)
- [What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation](https://mlsanity.com/paper/2406.16320/) (2024)
- [AdaptCLIP: Adapting CLIP for Universal Visual Anomaly Detection](https://mlsanity.com/paper/2505.09926/) (2025)
- [MedIAnomaly: A comparative study of anomaly detection in medical images](https://mlsanity.com/paper/2404.04518/) (2024)
- [MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model](https://mlsanity.com/paper/2406.11193/) (2024)
- [LogicAD: Explainable Anomaly Detection via VLM-based Text Feature Extraction](https://mlsanity.com/paper/2501.01767/) (2025)
- [SAE-V: Interpreting Multimodal Models for Enhanced Alignment](https://mlsanity.com/paper/2502.17514/) (2025)
- [Can Multimodal Large Language Models be Guided to Improve Industrial Anomaly Detection?](https://mlsanity.com/paper/2501.15795/) (2025)

## Built on by (most cited first, 3 of 3)

- [DV-Lens: Revealing the Functional Organization of Language Model Parameters](https://mlsanity.com/paper/2610.04489/) (2026)
- [Do correct answers have stable latent knowledge support?](https://mlsanity.com/paper/2610.15277/) (2026)
- [SuperAI: Merging Capabilities Across Heterogeneous Foundation Models](https://mlsanity.com/paper/2610.15322/) (2026)

## Abstract (arXiv)

Large-scale vision-language models (VLMs) exhibit remarkable zero-shot capabilities, yet the internal mechanisms driving their anomaly detection (AD) performance remain poorly understood. Current methods predominantly treat VLMs as black-box feature extractors, assuming that anomaly-specific knowledge must be acquired through external adapters or memory banks. In this paper, we challenge this assumption by arguing that anomaly knowledge is intrinsically embedded within pre-trained models but remains latent and under-activated. We hypothesize that this knowledge is concentrated within a sparse subset of anomaly-sensitive neurons. To validate this, we propose latent anomaly knowledge excavation (LAKE), a training-free framework that identifies and elicits these critical neuronal signals using only a minimal set of normal samples. By isolating these sensitive neurons, LAKE constructs a highly compact normality representation that integrates visual structural deviations with cross-modal semantic activations. Extensive experiments on industrial AD benchmarks demonstrate that LAKE achieves state-of-the-art performance while providing intrinsic, neuron-level interpretability. Ultimately, our work advocates for a paradigm shift: redefining anomaly detection as the targeted activation of latent pre-trained knowledge rather than the acquisition of a downstream task.