# Improving Vision Anomaly Detection with the Guidance of Language Modality

arXiv: https://arxiv.org/abs/2310.02821 · HTML: https://mlsanity.com/paper/2310.02821/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances](https://mlsanity.com/paper/2007.08176/) (2020)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [SSD: A Unified Framework for Self-Supervised Outlier Detection](https://mlsanity.com/paper/2103.12051/) (2021)
- [Fine-Grained Semantically Aligned Vision-Language Pre-Training](https://mlsanity.com/paper/2208.02515/) (2022)
- [Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learning](https://mlsanity.com/paper/2203.13049/) (2022)
- [Exploiting Epistemic Uncertainty of Anatomy Segmentation for Anomaly Detection in Retinal OCT](https://mlsanity.com/paper/1905.12806/) (2019)
- [Multi-Label Learning with Global and Local Label Correlation](https://mlsanity.com/paper/1704.01415/) (2017)
- [Exposing Outlier Exposure: What Can Be Learned From Few, One, and Zero Outlier Images](https://mlsanity.com/paper/2205.11474/) (2022)
- [Detection of Anomalies in Large Scale Accounting Data using Deep Autoencoder Networks](https://mlsanity.com/paper/1709.05254/) (2017)
- [Phonocardiographic Sensing using Deep Learning for Abnormal Heartbeat Detection](https://mlsanity.com/paper/1801.08322/) (2018)
- [Robust Meta-learning with Sampling Noise and Label Noise via Eigen-Reptile](https://mlsanity.com/paper/2206.01944/) (2022)
- [Cascaded Region-based Densely Connected Network for Event Detection: A Seismic Application](https://mlsanity.com/paper/1709.07943/) (2017)

## Built on by (most cited first, 7 of 7)

- [Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration](https://mlsanity.com/paper/2409.19872/) (2024)
- [FocusDiff: Advancing Fine-Grained Text-Image Alignment for Autoregressive Visual Generation through RL](https://mlsanity.com/paper/2506.05501/) (2025)
- [WiseEdit: Benchmarking Cognition- and Creativity-Informed Image Editing](https://mlsanity.com/paper/2512.00387/) (2025)
- [Anomaly Detection and Generation with Diffusion Models: A Survey](https://mlsanity.com/paper/2506.09368/) (2025)
- [KKA: Improving Vision Anomaly Detection through Anomaly-related Knowledge from Large Language Models](https://mlsanity.com/paper/2502.14880/) (2025)
- [Smile on the Face, Sadness in the Eyes: Bridging the Emotion Gap with a Multimodal Dataset of Eye and Facial Behaviors](https://mlsanity.com/paper/2512.16485/) (2025)
- [Neural-Collapse-guided Task-Free Continual Anomaly Detection](https://mlsanity.com/paper/2609.03406/) (2026)

## Abstract (arXiv)

Recent years have seen a surge of interest in anomaly detection for tackling industrial defect detection, event detection, etc. However, existing unsupervised anomaly detectors, particularly those for the vision modality, face significant challenges due to redundant information and sparse latent space. Conversely, the language modality performs well due to its relatively single data. This paper tackles the aforementioned challenges for vision modality from a multimodal point of view. Specifically, we propose Cross-modal Guidance (CMG), which consists of Cross-modal Entropy Reduction (CMER) and Cross-modal Linear Embedding (CMLE), to tackle the redundant information issue and sparse space issue, respectively. CMER masks parts of the raw image and computes the matching score with the text. Then, CMER discards irrelevant pixels to make the detector focus on critical contents. To learn a more compact latent space for the vision anomaly detector, CMLE learns a correlation structure matrix from the language modality, and then the latent space of vision modality will be learned with the guidance of the matrix. Thereafter, the vision latent space will get semantically similar images closer. Extensive experiments demonstrate the effectiveness of the proposed methods. Particularly, CMG outperforms the baseline that only uses images by 16.81%. Ablation experiments further confirm the synergy among the proposed methods, as each component depends on the other to achieve optimal performance.