# Fool Me Once: Robust Selective Segmentation via Out-of-Distribution Detection with Contrastive Learning

arXiv: https://arxiv.org/abs/2103.00869 · HTML: https://mlsanity.com/paper/2103.00869/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 43 references that are in the map.
- First posted on arXiv 1 Mar 2021.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances](https://mlsanity.com/paper/2007.08176/) (2020)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem](https://mlsanity.com/paper/1812.05720/) (2018)
- [ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1606.02147/) (2016)
- [Learning Memory-guided Normality for Anomaly Detection](https://mlsanity.com/paper/2003.13228/) (2020)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [Adversarially Learned One-Class Classifier for Novelty Detection](https://mlsanity.com/paper/1802.09088/) (2018)
- [Contrastive Training for Improved Out-of-Distribution Detection](https://mlsanity.com/paper/2007.05566/) (2020)
- [Ensemble Distribution Distillation](https://mlsanity.com/paper/1905.00076/) (2019)
- [Efficient GAN-Based Anomaly Detection](https://mlsanity.com/paper/1802.06222/) (2018)

## Built on by (most cited first, 9 of 9)

- [Out-of-Distribution Detection for LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2209.14435/) (2022)
- [Towards certifiable AI in aviation: landscape, challenges, and opportunities](https://mlsanity.com/paper/2409.08666/) (2024)
- [Out-of-Distribution Detection in LiDAR Semantic Segmentation Using Epistemic Uncertainty from Hierarchical GMMs](https://mlsanity.com/paper/2510.08631/) (2025)
- [Relative Energy Learning for LiDAR Out-of-Distribution Detection](https://mlsanity.com/paper/2511.06720/) (2025)
- [Masked Gamma-SSL: Learning Uncertainty Estimation via Masked Image Modeling](https://mlsanity.com/paper/2402.17622/) (2024)
- [Mitigating Distributional Shift in Semantic Segmentation via Uncertainty Estimation from Unlabelled Data](https://mlsanity.com/paper/2402.17653/) (2024)
- [DOODLER: Determining Out-Of-Distribution Likelihood from Encoder Reconstructions](https://mlsanity.com/paper/2109.13237/) (2021)
- [Anomaly-Aware Semantic Segmentation by Leveraging Synthetic-Unknown Data](https://mlsanity.com/paper/2111.14343/) (2021)
- [Elements of Active Continuous Learning and Uncertainty Self-Awareness: a Narrow Implementation for Face and Facial Expression Recognition](https://mlsanity.com/paper/2511.05574/) (2025)

## Abstract (arXiv)

In this work, we train a network to simultaneously perform segmentation and pixel-wise Out-of-Distribution (OoD) detection, such that the segmentation of unknown regions of scenes can be rejected. This is made possible by leveraging an OoD dataset with a novel contrastive objective and data augmentation scheme. By combining data including unknown classes in the training data, a more robust feature representation can be learned with known classes represented distinctly from those unknown. When presented with unknown classes or conditions, many current approaches for segmentation frequently exhibit high confidence in their inaccurate segmentations and cannot be trusted in many operational environments. We validate our system on a real-world dataset of unusual driving scenes, and show that by selectively segmenting scenes based on what is predicted as OoD, we can increase the segmentation accuracy by an IoU of 0.2 with respect to alternative techniques.