# Divide-and-Assemble: Learning Block-wise Memory for Unsupervised Anomaly Detection

arXiv: https://arxiv.org/abs/2107.13118 · HTML: https://mlsanity.com/paper/2107.13118/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 57 references that are in the map.
- First posted on arXiv 28 Jul 2021.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [GANomaly: Semi-Supervised Anomaly Detection via Adversarial Training](https://mlsanity.com/paper/1805.06725/) (2018)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [Sub-Image Anomaly Detection with Deep Pyramid Correspondences](https://mlsanity.com/paper/2005.02357/) (2020)
- [Learning Memory-guided Normality for Anomaly Detection](https://mlsanity.com/paper/2003.13228/) (2020)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [Learning Temporal Regularity in Video Sequences](https://mlsanity.com/paper/1604.04574/) (2016)
- [Key-Value Memory Networks for Directly Reading Documents](https://mlsanity.com/paper/1606.03126/) (2016)
- [Adversarially Learned One-Class Classifier for Novelty Detection](https://mlsanity.com/paper/1802.09088/) (2018)
- [OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representations](https://mlsanity.com/paper/1903.08550/) (2019)
- [Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering](https://mlsanity.com/paper/1904.04357/) (2019)
- [Efficient GAN-Based Anomaly Detection](https://mlsanity.com/paper/1802.06222/) (2018)
- [Modeling the Distribution of Normal Data in Pre-Trained Deep Features for Anomaly Detection](https://mlsanity.com/paper/2005.14140/) (2020)
- [MAST: A Memory-Augmented Self-supervised Tracker](https://mlsanity.com/paper/2002.07793/) (2020)
- [Skip-GANomaly: Skip Connected and Adversarially Trained Encoder-Decoder Anomaly Detection](https://mlsanity.com/paper/1901.08954/) (2019)
- [Abnormal Event Detection in Videos using Generative Adversarial Nets](https://mlsanity.com/paper/1708.09644/) (2017)
- [Anomaly Detection using One-Class Neural Networks](https://mlsanity.com/paper/1802.06360/) (2018)
- [MANTRA: Memory Augmented Networks for Multiple Trajectory Prediction](https://mlsanity.com/paper/2006.03340/) (2020)
- [Learning Dynamic Memory Networks for Object Tracking](https://mlsanity.com/paper/1803.07268/) (2018)
- [Improved anomaly detection by training an autoencoder with skip connections on images corrupted with Stain-shaped noise](https://mlsanity.com/paper/2008.12977/) (2020)
- [Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes](https://mlsanity.com/paper/1610.09027/) (2016)
- [Anomaly localization by modeling perceptual features](https://mlsanity.com/paper/2008.05369/) (2020)
- [Attribute Restoration Framework for Anomaly Detection](https://mlsanity.com/paper/1911.10676/) (2019)
- [Memory Augmented Generative Adversarial Networks for Anomaly Detection](https://mlsanity.com/paper/2002.02669/) (2020)

## Built on by (most cited first, 30 of 83)

- [Anomaly Detection via Reverse Distillation from One-Class Embedding](https://mlsanity.com/paper/2201.10703/) (2022)
- [A Unified Model for Multi-class Anomaly Detection](https://mlsanity.com/paper/2206.03687/) (2022)
- [SimpleNet: A Simple Network for Image Anomaly Detection and Localization](https://mlsanity.com/paper/2303.15140/) (2023)
- [Deep Industrial Image Anomaly Detection: A Survey](https://mlsanity.com/paper/2301.11514/) (2023)
- [Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection](https://mlsanity.com/paper/2403.12580/) (2024)
- [Catching Both Gray and Black Swans: Open-set Supervised Anomaly Detection](https://mlsanity.com/paper/2203.14506/) (2022)
- [Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Prompts](https://mlsanity.com/paper/2403.06495/) (2024)
- [BMAD: Benchmarks for Medical Anomaly Detection](https://mlsanity.com/paper/2306.11876/) (2023)
- [A Unified Anomaly Synthesis Strategy with Gradient Ascent for Industrial Anomaly Detection and Localization](https://mlsanity.com/paper/2407.09359/) (2024)
- [Deep Learning for Unsupervised Anomaly Localization in Industrial Images: A Survey](https://mlsanity.com/paper/2207.10298/) (2022)
- [IM-IAD: Industrial Image Anomaly Detection Benchmark in Manufacturing](https://mlsanity.com/paper/2301.13359/) (2023)
- [Segment Any Anomaly without Training via Hybrid Prompt Regularization](https://mlsanity.com/paper/2305.10724/) (2023)
- [Prototypical Residual Networks for Anomaly Detection and Localization](https://mlsanity.com/paper/2212.02031/) (2022)
- [SoftPatch: Unsupervised Anomaly Detection with Noisy Data](https://mlsanity.com/paper/2403.14233/) (2024)
- [Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2502.07601/) (2025)
- [DiffusionAD: Norm-guided One-step Denoising Diffusion for Anomaly Detection](https://mlsanity.com/paper/2303.08730/) (2023)
- [Multimodal Industrial Anomaly Detection by Crossmodal Feature Mapping](https://mlsanity.com/paper/2312.04521/) (2023)
- [Diversity-Measurable Anomaly Detection](https://mlsanity.com/paper/2303.05047/) (2023)
- [Weakly Supervised Video Anomaly Detection and Localization with Spatio-Temporal Prompts](https://mlsanity.com/paper/2408.05905/) (2024)
- [Anomaly Detection under Distribution Shift](https://mlsanity.com/paper/2303.13845/) (2023)
- [Focus the Discrepancy: Intra- and Inter-Correlation Learning for Image Anomaly Detection](https://mlsanity.com/paper/2308.02983/) (2023)
- [AdaptCLIP: Adapting CLIP for Universal Visual Anomaly Detection](https://mlsanity.com/paper/2505.09926/) (2025)
- [Anomaly Heterogeneity Learning for Open-set Supervised Anomaly Detection](https://mlsanity.com/paper/2310.12790/) (2023)
- [Target before Shooting: Accurate Anomaly Detection and Localization under One Millisecond via Cascade Patch Retrieval](https://mlsanity.com/paper/2308.06748/) (2023)
- [Fine-grained Abnormality Prompt Learning for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2410.10289/) (2024)
- [Visual Anomaly Detection Via Partition Memory Bank Module and Error Estimation](https://mlsanity.com/paper/2209.12441/) (2022)
- [Reconstruction from edge image combined with color and gradient difference for industrial surface anomaly detection](https://mlsanity.com/paper/2210.14485/) (2022)
- [Learning to Detect Multi-class Anomalies with Just One Normal Image Prompt](https://mlsanity.com/paper/2505.09264/) (2025)
- [Progressive Boundary Guided Anomaly Synthesis for Industrial Anomaly Detection](https://mlsanity.com/paper/2412.17458/) (2024)
- [FAPM: Fast Adaptive Patch Memory for Real-time Industrial Anomaly Detection](https://mlsanity.com/paper/2211.07381/) (2022)

## Abstract (arXiv)

Reconstruction-based methods play an important role in unsupervised anomaly detection in images. Ideally, we expect a perfect reconstruction for normal samples and poor reconstruction for abnormal samples. Since the generalizability of deep neural networks is difficult to control, existing models such as autoencoder do not work well. In this work, we interpret the reconstruction of an image as a divide-and-assemble procedure. Surprisingly, by varying the granularity of division on feature maps, we are able to modulate the reconstruction capability of the model for both normal and abnormal samples. That is, finer granularity leads to better reconstruction, while coarser granularity leads to poorer reconstruction. With proper granularity, the gap between the reconstruction error of normal and abnormal samples can be maximized. The divide-and-assemble framework is implemented by embedding a novel multi-scale block-wise memory module into an autoencoder network. Besides, we introduce adversarial learning and explore the semantic latent representation of the discriminator, which improves the detection of subtle anomaly. We achieve state-of-the-art performance on the challenging MVTec AD dataset. Remarkably, we improve the vanilla autoencoder model by 10.1% in terms of the AUROC score.