# Looking 3D: Anomaly Detection with 2D-3D Alignment

arXiv: https://arxiv.org/abs/2406.19393 · HTML: https://mlsanity.com/paper/2406.19393/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation](https://mlsanity.com/paper/2104.14812/) (2021)
- [The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation](https://mlsanity.com/paper/1904.03215/) (2019)
- [Anomaly Detection using One-Class Neural Networks](https://mlsanity.com/paper/1802.06360/) (2018)
- [KVT: k-NN Attention for Boosting Vision Transformers](https://mlsanity.com/paper/2106.00515/) (2021)
- [2D3D-MatchNet: Learning to Match Keypoints Across 2D Image and 3D Point Cloud](https://mlsanity.com/paper/1904.09742/) (2019)
- [Detecting semantic anomalies](https://mlsanity.com/paper/1908.04388/) (2019)
- [PAD: A Dataset and Benchmark for Pose-agnostic Anomaly Detection](https://mlsanity.com/paper/2310.07716/) (2023)
- [3D Pose Estimation and 3D Model Retrieval for Objects in the Wild](https://mlsanity.com/paper/1803.11493/) (2018)
- [LCD: Learned Cross-Domain Descriptors for 2D-3D Matching](https://mlsanity.com/paper/1911.09326/) (2019)
- [2D3D-MATR: 2D-3D Matching Transformer for Detection-free Registration between Images and Point Clouds](https://mlsanity.com/paper/2308.05667/) (2023)
- [PhotoShape: Photorealistic Materials for Large-Scale Shape Collections](https://mlsanity.com/paper/1809.09761/) (2018)
- [Location Field Descriptors: Single Image 3D Model Retrieval in the Wild](https://mlsanity.com/paper/1908.02853/) (2019)
- [DeepJoin: Learning a Joint Occupancy, Signed Distance, and Normal Field Function for Shape Repair](https://mlsanity.com/paper/2211.12400/) (2022)

## Built on by (most cited first, 13 of 13)

- [Look Inside for More: Internal Spatial Modality Perception for 3D Anomaly Detection](https://mlsanity.com/paper/2412.13461/) (2024)
- [Taming Anomalies with Down-Up Sampling Networks: Group Center Preserving Reconstruction for 3D Anomaly Detection](https://mlsanity.com/paper/2507.03903/) (2025)
- [A Lightweight 3D Anomaly Detection Method with Rotationally Invariant Features](https://mlsanity.com/paper/2511.13115/) (2025)
- [Odd-One-Out: Anomaly Detection by Comparing with Neighbors](https://mlsanity.com/paper/2406.20099/) (2024)
- [Examining the Source of Defects from a Mechanical Perspective for 3D Anomaly Detection](https://mlsanity.com/paper/2505.05901/) (2025)
- [Out-of-Distribution Semantic Occupancy Prediction](https://mlsanity.com/paper/2506.21185/) (2025)
- [Efficient Odd-One-Out Anomaly Detection](https://mlsanity.com/paper/2509.04326/) (2025)
- [Mentor3AD: Feature Reconstruction-based 3D Anomaly Detection via Multi-modality Mentor Learning](https://mlsanity.com/paper/2505.21420/) (2025)
- [BAAF: Universal Transformation of One-Class Classifiers for Unsupervised Image Anomaly Detection](https://mlsanity.com/paper/2602.13091/) (2026)
- [Open-Set Supervised 3D Anomaly Detection: An Industrial Dataset and a Generalisable Framework for Unknown Defects](https://mlsanity.com/paper/2604.01171/) (2026)
- [Learning Discriminative Signed Distance Functions from Multi-scale Level-of-detail Features for 3D Anomaly Detection](https://mlsanity.com/paper/2605.03437/) (2026)
- [MATCH: Flow Matching for Multi-View Anomaly Detection](https://mlsanity.com/paper/2606.24375/) (2026)
- [LTM: Large-scale Terrain Model for Wildfire-prone Landscapes](https://mlsanity.com/paper/2607.08711/) (2026)

## Abstract (arXiv)

Automatic anomaly detection based on visual cues holds practical significance in various domains, such as manufacturing and product quality assessment. This paper introduces a new conditional anomaly detection problem, which involves identifying anomalies in a query image by comparing it to a reference shape. To address this challenge, we have created a large dataset, BrokenChairs-180K, consisting of around 180K images, with diverse anomalies, geometries, and textures paired with 8,143 reference 3D shapes. To tackle this task, we have proposed a novel transformer-based approach that explicitly learns the correspondence between the query image and reference 3D shape via feature alignment and leverages a customized attention mechanism for anomaly detection. Our approach has been rigorously evaluated through comprehensive experiments, serving as a benchmark for future research in this domain.