# On Advantages of Mask-level Recognition for Outlier-aware Segmentation

arXiv: https://arxiv.org/abs/2301.03407 · HTML: https://mlsanity.com/paper/2301.03407/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 65 references that are in the map.
- First posted on arXiv 9 Jan 2023.

## Builds on (most cited first, 30 of 53)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [Why Normalizing Flows Fail to Detect Out-of-Distribution Data](https://mlsanity.com/paper/2006.08545/) (2020)
- [Input complexity and out-of-distribution detection with likelihood-based generative models](https://mlsanity.com/paper/1909.11480/) (2019)
- [Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1407.1808/) (2014)
- [Adversarial Reciprocal Points Learning for Open Set Recognition](https://mlsanity.com/paper/2103.00953/) (2021)
- [Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches](https://mlsanity.com/paper/1510.05970/) (2015)
- [SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation](https://mlsanity.com/paper/2104.14812/) (2021)

## Built on by (most cited first, 30 of 42)

- [Outlier detection by ensembling uncertainty with negative objectness](https://mlsanity.com/paper/2402.15374/) (2024)
- [PixOOD: Pixel-Level Out-of-Distribution Detection](https://mlsanity.com/paper/2405.19882/) (2024)
- [CSL: Class-Agnostic Structure-Constrained Learning for Segmentation Including the Unseen](https://mlsanity.com/paper/2312.05538/) (2023)
- [AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Driving](https://mlsanity.com/paper/2405.07865/) (2024)
- [Out-of-Distribution Segmentation in Autonomous Driving: Problems and State of the Art](https://mlsanity.com/paper/2503.08695/) (2025)
- [A Likelihood Ratio-Based Approach to Segmenting Unknown Objects](https://mlsanity.com/paper/2409.06424/) (2024)
- [Pixel-wise Gradient Uncertainty for Convolutional Neural Networks applied to Out-of-Distribution Segmentation](https://mlsanity.com/paper/2303.06920/) (2023)
- [Have We Ever Encountered This Before? Retrieving Out-of-Distribution Road Obstacles from Driving Scenes](https://mlsanity.com/paper/2309.04302/) (2023)
- [Diffusion for Out-of-Distribution Detection on Road Scenes and Beyond](https://mlsanity.com/paper/2407.15739/) (2024)
- [OoDIS: Anomaly Instance Segmentation and Detection Benchmark](https://mlsanity.com/paper/2406.11835/) (2024)
- [Exploring the Potential of World Models for Anomaly Detection in Autonomous Driving](https://mlsanity.com/paper/2308.05701/) (2023)
- [Prior2Former -- Evidential Modeling of Mask Transformers for Assumption-Free Open-World Panoptic Segmentation](https://mlsanity.com/paper/2504.04841/) (2025)
- [Anomaly-Aware Semantic Segmentation via Style-Aligned OoD Augmentation](https://mlsanity.com/paper/2308.09965/) (2023)
- [Finding Dino: A Plug-and-Play Framework for Zero-Shot Detection of Out-of-Distribution Objects Using Prototypes](https://mlsanity.com/paper/2404.07664/) (2024)
- [Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts](https://mlsanity.com/paper/2411.03829/) (2024)
- [Segmenting Objectiveness and Task-awareness Unknown Region for Autonomous Driving](https://mlsanity.com/paper/2504.19183/) (2025)
- [MC-PanDA: Mask Confidence for Panoptic Domain Adaptation](https://mlsanity.com/paper/2407.14110/) (2024)
- [VL4AD: Vision-Language Models Improve Pixel-wise Anomaly Detection](https://mlsanity.com/paper/2409.17330/) (2024)
- [TESGNN: Temporal Equivariant Scene Graph Neural Networks for Efficient and Robust Multi-View 3D Scene Understanding](https://mlsanity.com/paper/2411.10509/) (2024)
- [Segment-Level Road Obstacle Detection Using Visual Foundation Model Priors and Likelihood Ratios](https://mlsanity.com/paper/2412.05707/) (2024)
- [Multi-Scale Foreground-Background Confidence for Out-of-Distribution Segmentation](https://mlsanity.com/paper/2412.16990/) (2024)
- [Uncertainty-Aware Likelihood Ratio Estimation for Pixel-Wise Out-of-Distribution Detection](https://mlsanity.com/paper/2508.00587/) (2025)
- [Uncertainty estimates for semantic segmentation: providing enhanced reliability for automated motor claims handling](https://mlsanity.com/paper/2401.09245/) (2024)
- [FlowCLAS: Enhancing Normalizing Flow Via Contrastive Learning For Anomaly Segmentation](https://mlsanity.com/paper/2411.19888/) (2024)
- [Volumetric Mapping with Panoptic Refinement via Kernel Density Estimation for Mobile Robots](https://mlsanity.com/paper/2412.11241/) (2024)
- [Open-World Panoptic Segmentation](https://mlsanity.com/paper/2412.12740/) (2024)
- [Benchmarking Vision Foundation Models for Input Monitoring in Autonomous Driving](https://mlsanity.com/paper/2501.08083/) (2025)
- [A Dataset for Semantic Segmentation in the Presence of Unknowns](https://mlsanity.com/paper/2503.22309/) (2025)
- [Panoramic Out-of-Distribution Segmentation](https://mlsanity.com/paper/2505.03539/) (2025)
- [Out-of-Distribution Semantic Occupancy Prediction](https://mlsanity.com/paper/2506.21185/) (2025)

## Abstract (arXiv)

Most dense recognition approaches bring a separate decision in each particular pixel. These approaches deliver competitive performance in usual closed-set setups. However, important applications in the wild typically require strong performance in presence of outliers. We show that this demanding setup greatly benefit from mask-level predictions, even in the case of non-finetuned baseline models. Moreover, we propose an alternative formulation of dense recognition uncertainty that effectively reduces false positive responses at semantic borders. The proposed formulation produces a further improvement over a very strong baseline and sets the new state of the art in outlier-aware semantic segmentation with and without training on negative data. Our contributions also lead to performance improvement in a recent panoptic setup. In-depth experiments confirm that our approach succeeds due to implicit aggregation of pixel-level cues into mask-level predictions.