# Residual Pattern Learning for Pixel-wise Out-of-Distribution Detection in Semantic Segmentation

arXiv: https://arxiv.org/abs/2211.14512 · HTML: https://mlsanity.com/paper/2211.14512/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 51 references that are in the map.
- First posted on arXiv 26 Nov 2022.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [A Survey of Autonomous Driving: Common Practices and Emerging Technologies](https://mlsanity.com/paper/1906.05113/) (2019)
- [Exploring Cross-Image Pixel Contrast for Semantic Segmentation](https://mlsanity.com/paper/2101.11939/) (2021)
- [SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation](https://mlsanity.com/paper/2104.14812/) (2021)
- [Evaluating Bayesian Deep Learning Methods for Semantic Segmentation](https://mlsanity.com/paper/1811.12709/) (2018)
- [Detecting the Unexpected via Image Resynthesis](https://mlsanity.com/paper/1904.07595/) (2019)
- [Improving Semantic Segmentation via Video Propagation and Label Relaxation](https://mlsanity.com/paper/1812.01593/) (2018)
- [Entropy Maximization and Meta Classification for Out-Of-Distribution Detection in Semantic Segmentation](https://mlsanity.com/paper/2012.06575/) (2020)
- [Targeted Supervised Contrastive Learning for Long-Tailed Recognition](https://mlsanity.com/paper/2111.13998/) (2021)

## Built on by (most cited first, 30 of 55)

- [Learning Transferable Negative Prompts for Out-of-Distribution Detection](https://mlsanity.com/paper/2404.03248/) (2024)
- [Outlier detection by ensembling uncertainty with negative objectness](https://mlsanity.com/paper/2402.15374/) (2024)
- [Out-of-Distribution Detection in Long-Tailed Recognition with Calibrated Outlier Class Learning](https://mlsanity.com/paper/2312.10686/) (2023)
- [Segment Every Out-of-Distribution Object](https://mlsanity.com/paper/2311.16516/) (2023)
- [PixOOD: Pixel-Level Out-of-Distribution Detection](https://mlsanity.com/paper/2405.19882/) (2024)
- [CSL: Class-Agnostic Structure-Constrained Learning for Segmentation Including the Unseen](https://mlsanity.com/paper/2312.05538/) (2023)
- [Panoptic Out-of-Distribution Segmentation](https://mlsanity.com/paper/2310.11797/) (2023)
- [Placing Objects in Context via Inpainting for Out-of-distribution Segmentation](https://mlsanity.com/paper/2402.16392/) (2024)
- [Pixel-wise Gradient Uncertainty for Convolutional Neural Networks applied to Out-of-Distribution Segmentation](https://mlsanity.com/paper/2303.06920/) (2023)
- [Diffusion for Out-of-Distribution Detection on Road Scenes and Beyond](https://mlsanity.com/paper/2407.15739/) (2024)
- [Latency-aware Road Anomaly Segmentation in Videos: A Photorealistic Dataset and New Metrics](https://mlsanity.com/paper/2401.04942/) (2024)
- [Conformal Performance Range Prediction for Segmentation Output Quality Control](https://mlsanity.com/paper/2407.13307/) (2024)
- [Extremely Simple Multimodal Outlier Synthesis for Out-of-Distribution Detection and Segmentation](https://mlsanity.com/paper/2505.16985/) (2025)
- [OpenCIL: Benchmarking Out-of-Distribution Detection in Class-Incremental Learning](https://mlsanity.com/paper/2407.06045/) (2024)
- [Anomaly-Aware Semantic Segmentation via Style-Aligned OoD Augmentation](https://mlsanity.com/paper/2308.09965/) (2023)
- [Zero-Shot Out-of-Distribution Detection with Outlier Label Exposure](https://mlsanity.com/paper/2406.01170/) (2024)
- [ItTakesTwo: Leveraging Peer Representations for Semi-supervised LiDAR Semantic Segmentation](https://mlsanity.com/paper/2407.07171/) (2024)
- [Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts](https://mlsanity.com/paper/2411.03829/) (2024)
- [Segmenting Objectiveness and Task-awareness Unknown Region for Autonomous Driving](https://mlsanity.com/paper/2504.19183/) (2025)
- [Long-Tailed Out-of-Distribution Detection via Normalized Outlier Distribution Adaptation](https://mlsanity.com/paper/2410.20807/) (2024)
- [Open-Vocabulary Domain Generalization in Urban-Scene Segmentation](https://mlsanity.com/paper/2602.18853/) (2026)
- [UMAD: Unsupervised Mask-Level Anomaly Detection for Autonomous Driving](https://mlsanity.com/paper/2406.06370/) (2024)
- [Multi-Scale Foreground-Background Confidence for Out-of-Distribution Segmentation](https://mlsanity.com/paper/2412.16990/) (2024)
- [Mitigating the Modality Gap: Few-Shot Out-of-Distribution Detection with Multi-modal Prototypes and Image Bias Estimation](https://mlsanity.com/paper/2502.00662/) (2025)
- [Uncertainty-Aware Likelihood Ratio Estimation for Pixel-Wise Out-of-Distribution Detection](https://mlsanity.com/paper/2508.00587/) (2025)
- [Self-Supervised Likelihood Estimation with Energy Guidance for Anomaly Segmentation in Urban Scenes](https://mlsanity.com/paper/2302.06815/) (2023)
- [FlowCLAS: Enhancing Normalizing Flow Via Contrastive Learning For Anomaly Segmentation](https://mlsanity.com/paper/2411.19888/) (2024)
- [Benchmarking Vision Foundation Models for Input Monitoring in Autonomous Driving](https://mlsanity.com/paper/2501.08083/) (2025)
- [Panoramic Out-of-Distribution Segmentation](https://mlsanity.com/paper/2505.03539/) (2025)
- [Anomalies by Synthesis: Anomaly Detection using Generative Diffusion Models for Off-Road Navigation](https://mlsanity.com/paper/2505.22805/) (2025)

## Abstract (arXiv)

Semantic segmentation models classify pixels into a set of known (``in-distribution'') visual classes. When deployed in an open world, the reliability of these models depends on their ability not only to classify in-distribution pixels but also to detect out-of-distribution (OoD) pixels. Historically, the poor OoD detection performance of these models has motivated the design of methods based on model re-training using synthetic training images that include OoD visual objects. Although successful, these re-trained methods have two issues: 1) their in-distribution segmentation accuracy may drop during re-training, and 2) their OoD detection accuracy does not generalise well to new contexts (e.g., country surroundings) outside the training set (e.g., city surroundings). In this paper, we mitigate these issues with: (i) a new residual pattern learning (RPL) module that assists the segmentation model to detect OoD pixels without affecting the inlier segmentation performance; and (ii) a novel context-robust contrastive learning (CoroCL) that enforces RPL to robustly detect OoD pixels among various contexts. Our approach improves by around 10% FPR and 7% AuPRC the previous state-of-the-art in Fishyscapes, Segment-Me-If-You-Can, and RoadAnomaly datasets. Our code is available at: https://github.com/yyliu01/RPL.