# What Memory Composition Does Not Tell Us About Anomaly Detection

arXiv: https://arxiv.org/abs/2608.23295 · HTML: https://mlsanity.com/paper/2608.23295/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Builds on (most cited first, 18 of 18)

- [Towards Total Recall in Industrial Anomaly Detection](https://mlsanity.com/paper/2106.08265/) (2021)
- [PaDiM: a Patch Distribution Modeling Framework for Anomaly Detection and Localization](https://mlsanity.com/paper/2011.08785/) (2020)
- [SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2207.14315/) (2022)
- [Sub-Image Anomaly Detection with Deep Pyramid Correspondences](https://mlsanity.com/paper/2005.02357/) (2020)
- [GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training](https://mlsanity.com/paper/2103.00123/) (2021)
- [Coresets via Bilevel Optimization for Continual Learning and Streaming](https://mlsanity.com/paper/2006.03875/) (2020)
- [PANDA: Adapting Pretrained Features for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2010.05903/) (2020)
- [Mean-Shifted Contrastive Loss for Anomaly Detection](https://mlsanity.com/paper/2106.03844/) (2021)
- [SoftPatch: Unsupervised Anomaly Detection with Noisy Data](https://mlsanity.com/paper/2403.14233/) (2024)
- [Latent Outlier Exposure for Anomaly Detection with Contaminated Data](https://mlsanity.com/paper/2202.08088/) (2022)
- [Constant Approximation for k-Median and k-Means with Outliers via Iterative Rounding](https://mlsanity.com/paper/1711.01323/) (2017)
- [Self-supervise, Refine, Repeat: Improving Unsupervised Anomaly Detection](https://mlsanity.com/paper/2106.06115/) (2021)
- [FUN-AD: Fully Unsupervised Learning for Anomaly Detection with Noisy Training Data](https://mlsanity.com/paper/2411.16110/) (2024)
- [SoftPatch+: Fully Unsupervised Anomaly Classification and Segmentation](https://mlsanity.com/paper/2412.20870/) (2024)
- [Data refinement for fully unsupervised visual inspection using pre-trained networks](https://mlsanity.com/paper/2202.12759/) (2022)
- [A Synergy Scoring Filter for Unsupervised Anomaly Detection with Noisy Data](https://mlsanity.com/paper/2502.13992/) (2025)
- [Leveraging Learning Bias for Noisy Anomaly Detection](https://mlsanity.com/paper/2508.07441/) (2025)
- [Memory-Distilled Selection for Noise-Robust Anomaly Detection](https://mlsanity.com/paper/2605.26676/) (2026)

## Abstract (arXiv)

Memory-based anomaly detectors store nominal training patches and score test patches against this memory. A patch selected for coverage therefore becomes a nor- mal reference without a separate check that geometric rarity makes it safe to trust. We probe this coupling with sparse training contamination. Under fixed representa- tions and memory budgets, we compare random, medoid, local, and global coverage selectors. We then use CLEANCON, an out-of-bag cross-image support gate that changes candidate-image eligibility while fixing the representation, absolute mem- ory size, builder, and inference rule. Global coverage strongly over-represents sparse contamination. CLEANCON reduces final-memory contamination to approx- imately zero and increases category-macro P-AP in all 12 matched comparisons. Yet along a retention sweep, the lowest-contamination memory does not attain the highest P-AP; performance continues to improve while contamination rises. Mem- ory contamination therefore does not order the resulting memories by P-AP