# AnomalyControl: Learning Cross-modal Semantic Features for Controllable Anomaly Synthesis

arXiv: https://arxiv.org/abs/2412.06510 · HTML: https://mlsanity.com/paper/2412.06510/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 9 Dec 2024.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [DRAEM -- A discriminatively trained reconstruction embedding for surface anomaly detection](https://mlsanity.com/paper/2108.07610/) (2021)
- [Differentiable Augmentation for Data-Efficient GAN Training](https://mlsanity.com/paper/2006.10738/) (2020)
- [Training-Free Layout Control with Cross-Attention Guidance](https://mlsanity.com/paper/2304.03373/) (2023)
- [Few-shot Image Generation via Cross-domain Correspondence](https://mlsanity.com/paper/2104.06820/) (2021)
- [Deep Industrial Image Anomaly Detection: A Survey](https://mlsanity.com/paper/2301.11514/) (2023)
- [RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths](https://mlsanity.com/paper/2305.18295/) (2023)
- [RealNet: A Feature Selection Network with Realistic Synthetic Anomaly for Anomaly Detection](https://mlsanity.com/paper/2403.05897/) (2024)
- [AnomalyDiffusion: Few-Shot Anomaly Image Generation with Diffusion Model](https://mlsanity.com/paper/2312.05767/) (2023)
- [Defect-GAN: High-Fidelity Defect Synthesis for Automated Defect Inspection](https://mlsanity.com/paper/2103.15158/) (2021)
- [Few-Shot Defect Image Generation via Defect-Aware Feature Manipulation](https://mlsanity.com/paper/2303.02389/) (2023)
- [EasyNet: An Easy Network for 3D Industrial Anomaly Detection](https://mlsanity.com/paper/2307.13925/) (2023)
- [AnomalyXFusion: Multi-modal Anomaly Synthesis with Diffusion](https://mlsanity.com/paper/2404.19444/) (2024)

## Built on by (most cited first, 7 of 7)

- [A Survey on Industrial Anomaly Synthesis](https://mlsanity.com/paper/2502.16412/) (2025)
- [AnomalyPainter: Vision-Language-Diffusion Synergy for Zero-Shot Realistic and Diverse Industrial Anomaly Synthesis](https://mlsanity.com/paper/2503.07253/) (2025)
- [NexViTAD: Few-shot Unsupervised Cross-Domain Defect Detection via Vision Foundation Models and Multi-Task Learning](https://mlsanity.com/paper/2507.07579/) (2025)
- [Training-Free Anomaly Generation via Dual-Attention Enhancement in Diffusion Model](https://mlsanity.com/paper/2508.11550/) (2025)
- [FAST: Foreground-aware Diffusion with Accelerated Sampling Trajectory for Segmentation-oriented Anomaly Synthesis](https://mlsanity.com/paper/2509.20295/) (2025)
- [Achieving Fine-grained Cross-modal Understanding through Brain-inspired Hierarchical Representation Learning](https://mlsanity.com/paper/2601.01339/) (2026)
- [DeCo: Zero-Shot Industrial Anomaly Generation through Decoupling and Recoupling](https://mlsanity.com/paper/2608.07904/) (2026)

## Abstract (arXiv)

Anomaly synthesis is a crucial approach to augment abnormal data for advancing anomaly inspection. Based on the knowledge from the large-scale pre-training, existing text-to-image anomaly synthesis methods predominantly focus on textual information or coarse-aligned visual features to guide the entire generation process. However, these methods often lack sufficient descriptors to capture the complicated characteristics of realistic anomalies (e.g., the fine-grained visual pattern of anomalies), limiting the realism and generalization of the generation process. To this end, we propose a novel anomaly synthesis framework called AnomalyControl to learn cross-modal semantic features as guidance signals, which could encode the generalized anomaly cues from text-image reference prompts and improve the realism of synthesized abnormal samples. Specifically, AnomalyControl adopts a flexible and non-matching prompt pair (i.e., a text-image reference prompt and a targeted text prompt), where a Cross-modal Semantic Modeling (CSM) module is designed to extract cross-modal semantic features from the textual and visual descriptors. Then, an Anomaly-Semantic Enhanced Attention (ASEA) mechanism is formulated to allow CSM to focus on the specific visual patterns of the anomaly, thus enhancing the realism and contextual relevance of the generated anomaly features. Treating cross-modal semantic features as the prior, a Semantic Guided Adapter (SGA) is designed to encode effective guidance signals for the adequate and controllable synthesis process. Extensive experiments indicate that AnomalyControl can achieve state-of-the-art results in anomaly synthesis compared with existing methods while exhibiting superior performance for downstream tasks.