# Mean-Shifted Contrastive Loss for Anomaly Detection

arXiv: https://arxiv.org/abs/2106.03844 · HTML: https://mlsanity.com/paper/2106.03844/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 42 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 24 of 24)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances](https://mlsanity.com/paper/2007.08176/) (2020)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmark](https://mlsanity.com/paper/1909.00133/) (2019)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [Learning Representations for Automatic Colorization](https://mlsanity.com/paper/1603.06668/) (2016)
- [Classification-Based Anomaly Detection for General Data](https://mlsanity.com/paper/2005.02359/) (2020)
- [What makes ImageNet good for transfer learning?](https://mlsanity.com/paper/1608.08614/) (2016)
- [PANDA: Adapting Pretrained Features for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2010.05903/) (2020)
- [Learning and Evaluating Representations for Deep One-class Classification](https://mlsanity.com/paper/2011.02578/) (2020)
- [Learning Deep Features for One-Class Classification](https://mlsanity.com/paper/1801.05365/) (2018)
- [Detecting semantic anomalies](https://mlsanity.com/paper/1908.04388/) (2019)
- [Rethinking Assumptions in Deep Anomaly Detection](https://mlsanity.com/paper/2006.00339/) (2020)

## Built on by (most cited first, 30 of 88)

- [SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2207.14315/) (2022)
- [AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2310.18961/) (2023)
- [Deep Industrial Image Anomaly Detection: A Survey](https://mlsanity.com/paper/2301.11514/) (2023)
- [Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2502.07601/) (2025)
- [Focus the Discrepancy: Intra- and Inter-Correlation Learning for Image Anomaly Detection](https://mlsanity.com/paper/2308.02983/) (2023)
- [GeneralAD: Anomaly Detection Across Domains by Attending to Distorted Features](https://mlsanity.com/paper/2407.12427/) (2024)
- [PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection](https://mlsanity.com/paper/2410.00320/) (2024)
- [Heterogeneous Contrastive Learning for Foundation Models and Beyond](https://mlsanity.com/paper/2404.00225/) (2024)
- [MedIAnomaly: A comparative study of anomaly detection in medical images](https://mlsanity.com/paper/2404.04518/) (2024)
- [Transformaly -- Two (Feature Spaces) Are Better Than One](https://mlsanity.com/paper/2112.04185/) (2021)
- [Self-supervised Pseudo Multi-class Pre-training for Unsupervised Anomaly Detection and Segmentation in Medical Images](https://mlsanity.com/paper/2109.01303/) (2021)
- [A Survey on Unsupervised Anomaly Detection Algorithms for Industrial Images](https://mlsanity.com/paper/2204.11161/) (2022)
- [Fr'echet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasets](https://mlsanity.com/paper/2412.01496/) (2024)
- [Beyond Individual Input for Deep Anomaly Detection on Tabular Data](https://mlsanity.com/paper/2305.15121/) (2023)
- [Anomaly Detection Requires Better Representations](https://mlsanity.com/paper/2210.10773/) (2022)
- [Detection of out-of-distribution samples using binary neuron activation patterns](https://mlsanity.com/paper/2212.14268/) (2022)
- [Set Features for Fine-grained Anomaly Detection](https://mlsanity.com/paper/2302.12245/) (2023)
- [A Dual-way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking](https://mlsanity.com/paper/2312.11816/) (2023)
- [Unsupervised Outlier Detection using Memory and Contrastive Learning](https://mlsanity.com/paper/2107.12642/) (2021)
- [Anomalies, Representations, and Self-Supervision](https://mlsanity.com/paper/2301.04660/) (2023)
- [CtxMIM: Context-Enhanced Masked Image Modeling for Remote Sensing Image Understanding](https://mlsanity.com/paper/2310.00022/) (2023)
- [Universal Novelty Detection Through Adaptive Contrastive Learning](https://mlsanity.com/paper/2408.10798/) (2024)
- [Invariant Anomaly Detection under Distribution Shifts: A Causal Perspective](https://mlsanity.com/paper/2312.14329/) (2023)
- [No Free Lunch: The Hazards of Over-Expressive Representations in Anomaly Detection](https://mlsanity.com/paper/2306.07284/) (2023)
- [RODEO: Robust Outlier Detection via Exposing Adaptive Out-of-Distribution Samples](https://mlsanity.com/paper/2501.16971/) (2025)
- [AF-CLIP: Zero-Shot Anomaly Detection via Anomaly-Focused CLIP Adaptation](https://mlsanity.com/paper/2507.19949/) (2025)
- [Efficient Anomaly Detection Using Self-Supervised Multi-Cue Tasks](https://mlsanity.com/paper/2111.12379/) (2021)
- [Retrieval Augmented Deep Anomaly Detection for Tabular Data](https://mlsanity.com/paper/2401.17052/) (2024)
- [Data Invariants to Understand Unsupervised Out-of-Distribution Detection](https://mlsanity.com/paper/2111.13362/) (2021)
- [Out-of-Distribution Detection Without Class Labels](https://mlsanity.com/paper/2112.07662/) (2021)

## Abstract (arXiv)

Deep anomaly detection methods learn representations that separate between normal and anomalous images. Although self-supervised representation learning is commonly used, small dataset sizes limit its effectiveness. It was previously shown that utilizing external, generic datasets (e.g. ImageNet classification) can significantly improve anomaly detection performance. One approach is outlier exposure, which fails when the external datasets do not resemble the anomalies. We take the approach of transferring representations pre-trained on external datasets for anomaly detection. Anomaly detection performance can be significantly improved by fine-tuning the pre-trained representations on the normal training images. In this paper, we first demonstrate and analyze that contrastive learning, the most popular self-supervised learning paradigm cannot be naively applied to pre-trained features. The reason is that pre-trained feature initialization causes poor conditioning for standard contrastive objectives, resulting in bad optimization dynamics. Based on our analysis, we provide a modified contrastive objective, the Mean-Shifted Contrastive Loss. Our method is highly effective and achieves a new state-of-the-art anomaly detection performance including 98.6% ROC-AUC on the CIFAR-10 dataset.