# Pixel-wise Anomaly Detection in Complex Driving Scenes

arXiv: https://arxiv.org/abs/2103.05445 · HTML: https://mlsanity.com/paper/2103.05445/

## Key facts

- Cited by 109 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 41 references that are in the map.
- First posted on arXiv 9 Mar 2021.

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [ICNet for Real-Time Semantic Segmentation on High-Resolution Images](https://mlsanity.com/paper/1704.08545/) (2017)
- [Generating Images with Perceptual Similarity Metrics based on Deep Networks](https://mlsanity.com/paper/1602.02644/) (2016)
- [Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding](https://mlsanity.com/paper/1511.02680/) (2015)
- [Deep Autoencoding Models for Unsupervised Anomaly Segmentation in Brain MR Images](https://mlsanity.com/paper/1804.04488/) (2018)
- [Evaluating Bayesian Deep Learning Methods for Semantic Segmentation](https://mlsanity.com/paper/1811.12709/) (2018)
- [Detecting the Unexpected via Image Resynthesis](https://mlsanity.com/paper/1904.07595/) (2019)
- [Improving Semantic Segmentation via Video Propagation and Label Relaxation](https://mlsanity.com/paper/1812.01593/) (2018)
- [The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation](https://mlsanity.com/paper/1904.03215/) (2019)
- [Lost and Found: Detecting Small Road Hazards for Self-Driving Vehicles](https://mlsanity.com/paper/1609.04653/) (2016)
- [Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis](https://mlsanity.com/paper/1910.06809/) (2019)
- [Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation](https://mlsanity.com/paper/2003.08440/) (2020)
- [Prediction Error Meta Classification in Semantic Segmentation: Detection via Aggregated Dispersion Measures of Softmax Probabilities](https://mlsanity.com/paper/1811.00648/) (2018)

## Built on by (most cited first, 30 of 109)

- [SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation](https://mlsanity.com/paper/2104.14812/) (2021)
- [Catching Both Gray and Black Swans: Open-set Supervised Anomaly Detection](https://mlsanity.com/paper/2203.14506/) (2022)
- [GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models](https://mlsanity.com/paper/2210.02025/) (2022)
- [Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes](https://mlsanity.com/paper/2111.12264/) (2021)
- [RbA: Segmenting Unknown Regions Rejected by All](https://mlsanity.com/paper/2211.14293/) (2022)
- [Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models](https://mlsanity.com/paper/2505.23757/) (2025)
- [Residual Pattern Learning for Pixel-wise Out-of-Distribution Detection in Semantic Segmentation](https://mlsanity.com/paper/2211.14512/) (2022)
- [Unmasking Anomalies in Road-Scene Segmentation](https://mlsanity.com/paper/2307.13316/) (2023)
- [DenseHybrid: Hybrid Anomaly Detection for Dense Open-set Recognition](https://mlsanity.com/paper/2207.02606/) (2022)
- [On Advantages of Mask-level Recognition for Outlier-aware Segmentation](https://mlsanity.com/paper/2301.03407/) (2023)
- [Maskomaly:Zero-Shot Mask Anomaly Segmentation](https://mlsanity.com/paper/2305.16972/) (2023)
- [Outlier detection by ensembling uncertainty with negative objectness](https://mlsanity.com/paper/2402.15374/) (2024)
- [Hierarchical Gaussian Mixture Normalizing Flow Modeling for Unified Anomaly Detection](https://mlsanity.com/paper/2403.13349/) (2024)
- [Feature Generation and Hypothesis Verification for Reliable Face Anti-Spoofing](https://mlsanity.com/paper/2112.14894/) (2021)
- [BatchNorm-based Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2311.15367/) (2023)
- [Segment Every Out-of-Distribution Object](https://mlsanity.com/paper/2311.16516/) (2023)
- [PixOOD: Pixel-Level Out-of-Distribution Detection](https://mlsanity.com/paper/2405.19882/) (2024)
- [Two Video Data Sets for Tracking and Retrieval of Out of Distribution Objects](https://mlsanity.com/paper/2210.02074/) (2022)
- [Mask2Anomaly: Mask Transformer for Universal Open-set Segmentation](https://mlsanity.com/paper/2309.04573/) (2023)
- [Balanced Energy Regularization Loss for Out-of-distribution Detection](https://mlsanity.com/paper/2306.10485/) (2023)
- [UGainS: Uncertainty Guided Anomaly Instance Segmentation](https://mlsanity.com/paper/2308.02046/) (2023)
- [CSL: Class-Agnostic Structure-Constrained Learning for Segmentation Including the Unseen](https://mlsanity.com/paper/2312.05538/) (2023)
- [ATTA: Anomaly-aware Test-Time Adaptation for Out-of-Distribution Detection in Segmentation](https://mlsanity.com/paper/2309.05994/) (2023)
- [AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Driving](https://mlsanity.com/paper/2405.07865/) (2024)
- [Hyperdimensional Feature Fusion for Out-Of-Distribution Detection](https://mlsanity.com/paper/2112.05341/) (2021)
- [Latent-OFER: Detect, Mask, and Reconstruct with Latent Vectors for Occluded Facial Expression Recognition](https://mlsanity.com/paper/2307.11404/) (2023)
- [Out-of-Distribution Segmentation in Autonomous Driving: Problems and State of the Art](https://mlsanity.com/paper/2503.08695/) (2025)
- [Placing Objects in Context via Inpainting for Out-of-distribution Segmentation](https://mlsanity.com/paper/2402.16392/) (2024)
- [A Survey of Visual Sensory Anomaly Detection](https://mlsanity.com/paper/2202.07006/) (2022)
- [Pixel-wise Gradient Uncertainty for Convolutional Neural Networks applied to Out-of-Distribution Segmentation](https://mlsanity.com/paper/2303.06920/) (2023)

## Abstract (arXiv)

The inability of state-of-the-art semantic segmentation methods to detect anomaly instances hinders them from being deployed in safety-critical and complex applications, such as autonomous driving. Recent approaches have focused on either leveraging segmentation uncertainty to identify anomalous areas or re-synthesizing the image from the semantic label map to find dissimilarities with the input image. In this work, we demonstrate that these two methodologies contain complementary information and can be combined to produce robust predictions for anomaly segmentation. We present a pixel-wise anomaly detection framework that uses uncertainty maps to improve over existing re-synthesis methods in finding dissimilarities between the input and generated images. Our approach works as a general framework around already trained segmentation networks, which ensures anomaly detection without compromising segmentation accuracy, while significantly outperforming all similar methods. Top-2 performance across a range of different anomaly datasets shows the robustness of our approach to handling different anomaly instances.