# Pseudo Outlier Exposure for Out-of-Distribution Detection using Pretrained Transformers

arXiv: https://arxiv.org/abs/2307.09455 · HTML: https://mlsanity.com/paper/2307.09455/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [ReAct: Out-of-distribution Detection With Rectified Activations](https://mlsanity.com/paper/2111.12797/) (2021)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data](https://mlsanity.com/paper/2002.11297/) (2020)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [DICE: Leveraging Sparsification for Out-of-Distribution Detection](https://mlsanity.com/paper/2111.09805/) (2021)
- [Types of Out-of-Distribution Texts and How to Detect Them](https://mlsanity.com/paper/2109.06827/) (2021)
- [Contrastive Out-of-Distribution Detection for Pretrained Transformers](https://mlsanity.com/paper/2104.08812/) (2021)
- [Pre-Training Transformers as Energy-Based Cloze Models](https://mlsanity.com/paper/2012.08561/) (2020)
- [News Category Dataset](https://mlsanity.com/paper/2209.11429/) (2022)
- [Interactive Language Learning by Question Answering](https://mlsanity.com/paper/1908.10909/) (2019)

## Built on by (most cited first, 5 of 5)

- [Rethinking the Uncertainty: A Critical Review and Analysis in the Era of Large Language Models](https://mlsanity.com/paper/2410.20199/) (2024)
- [Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers](https://mlsanity.com/paper/2510.18358/) (2025)
- [Torch-Uncertainty: A Deep Learning Framework for Uncertainty Quantification](https://mlsanity.com/paper/2511.10282/) (2025)
- [Out-of-Distribution Detection through Soft Clustering with Non-Negative Kernel Regression](https://mlsanity.com/paper/2407.13141/) (2024)
- [Semantic Flip: Synthetic OOD Generation for Robust Refusal in Embodied Question Answering and Spatial Localization](https://mlsanity.com/paper/2606.16898/) (2026)

## Abstract (arXiv)

For real-world language applications, detecting an out-of-distribution (OOD) sample is helpful to alert users or reject such unreliable samples. However, modern over-parameterized language models often produce overconfident predictions for both in-distribution (ID) and OOD samples. In particular, language models suffer from OOD samples with a similar semantic representation to ID samples since these OOD samples lie near the ID manifold. A rejection network can be trained with ID and diverse outlier samples to detect test OOD samples, but explicitly collecting auxiliary OOD datasets brings an additional burden for data collection. In this paper, we propose a simple but effective method called Pseudo Outlier Exposure (POE) that constructs a surrogate OOD dataset by sequentially masking tokens related to ID classes. The surrogate OOD sample introduced by POE shows a similar representation to ID data, which is most effective in training a rejection network. Our method does not require any external OOD data and can be easily implemented within off-the-shelf Transformers. A comprehensive comparison with state-of-the-art algorithms demonstrates POE's competitiveness on several text classification benchmarks.