# Puzzle-AE: Novelty Detection in Images through Solving Puzzles

arXiv: https://arxiv.org/abs/2008.12959 · HTML: https://mlsanity.com/paper/2008.12959/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2020.

## Built on by (most cited first, 23 of 23)

- [CutPaste: Self-Supervised Learning for Anomaly Detection and Localization](https://mlsanity.com/paper/2104.04015/) (2021)
- [A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges](https://mlsanity.com/paper/2110.14051/) (2021)
- [Omni-frequency Channel-selection Representations for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2203.00259/) (2022)
- [Transformaly -- Two (Feature Spaces) Are Better Than One](https://mlsanity.com/paper/2112.04185/) (2021)
- [Adversarially Robust One-class Novelty Detection](https://mlsanity.com/paper/2108.11168/) (2021)
- [A Survey on Unsupervised Anomaly Detection Algorithms for Industrial Images](https://mlsanity.com/paper/2204.11161/) (2022)
- [Visual Anomaly Detection Via Partition Memory Bank Module and Error Estimation](https://mlsanity.com/paper/2209.12441/) (2022)
- [Visual Anomaly Detection for Images: A Survey](https://mlsanity.com/paper/2109.13157/) (2021)
- [PEDENet: Image Anomaly Localization via Patch Embedding and Density Estimation](https://mlsanity.com/paper/2110.15525/) (2021)
- [Precursor-of-Anomaly Detection for Irregular Time Series](https://mlsanity.com/paper/2306.15489/) (2023)
- [Efficient Anomaly Detection Using Self-Supervised Multi-Cue Tasks](https://mlsanity.com/paper/2111.12379/) (2021)
- [Data Invariants to Understand Unsupervised Out-of-Distribution Detection](https://mlsanity.com/paper/2111.13362/) (2021)
- [ADPS: Asymmetric Distillation Post-Segmentation for Image Anomaly Detection](https://mlsanity.com/paper/2210.10495/) (2022)
- [Ano-Graph: Learning Normal Scene Contextual Graphs to Detect Video Anomalies](https://mlsanity.com/paper/2103.10502/) (2021)
- [Fake It Till You Make It: Towards Accurate Near-Distribution Novelty Detection](https://mlsanity.com/paper/2205.14297/) (2022)
- [Fine-grained Anomaly Detection via Multi-task Self-Supervision](https://mlsanity.com/paper/2104.09993/) (2021)
- [A Recover-then-Discriminate Framework for Robust Anomaly Detection](https://mlsanity.com/paper/2406.04608/) (2024)
- [Multi-feature Reconstruction Network using Crossed-mask Restoration for Unsupervised Industrial Anomaly Detection](https://mlsanity.com/paper/2404.13273/) (2024)
- [Deep Learning-Enhanced Visual Monitoring in Hazardous Underwater Environments with a Swarm of Micro-Robots](https://mlsanity.com/paper/2503.02752/) (2025)
- [InForecaster: Forecasting Influenza Hemagglutinin Mutations Through the Lens of Anomaly Detection](https://mlsanity.com/paper/2210.13709/) (2022)
- [Image Outlier Detection Without Training using RANSAC](https://mlsanity.com/paper/2307.12301/) (2023)
- [KDC-MAE: Knowledge Distilled Contrastive Mask Auto-Encoder](https://mlsanity.com/paper/2411.12270/) (2024)
- [Redefining Normal: A Novel Object-Level Approach for Multi-Object Novelty Detection](https://mlsanity.com/paper/2412.11148/) (2024)

## Abstract (arXiv)

Autoencoder, as an essential part of many anomaly detection methods, is lacking flexibility on normal data in complex datasets. U-Net is proved to be effective for this purpose but overfits on the training data if trained by just using reconstruction error similar to other AE-based frameworks. Puzzle-solving, as a pretext task of self-supervised learning (SSL) methods, has earlier proved its ability in learning semantically meaningful features. We show that training U-Nets based on this task is an effective remedy that prevents overfitting and facilitates learning beyond pixel-level features. Shortcut solutions, however, are a big challenge in SSL tasks, including jigsaw puzzles. We propose adversarial robust training as an effective automatic shortcut removal. We achieve competitive or superior results compared to the State of the Art (SOTA) anomaly detection methods on various toy and real-world datasets. Unlike many competitors, the proposed framework is stable, fast, data-efficient, and does not require unprincipled early stopping.