# Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detection

arXiv: https://arxiv.org/abs/2305.13282 · HTML: https://mlsanity.com/paper/2305.13282/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 61 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 30 of 49)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [ReAct: Out-of-distribution Detection With Rectified Activations](https://mlsanity.com/paper/2111.12797/) (2021)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Calibration of Pre-trained Transformers](https://mlsanity.com/paper/2003.07892/) (2020)
- [Pretrained Transformers Improve Out-of-Distribution Robustness](https://mlsanity.com/paper/2004.06100/) (2020)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [Mitigating Neural Network Overconfidence with Logit Normalization](https://mlsanity.com/paper/2205.09310/) (2022)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [Selective Question Answering under Domain Shift](https://mlsanity.com/paper/2006.09462/) (2020)
- [Delving into Out-of-Distribution Detection with Vision-Language Representations](https://mlsanity.com/paper/2211.13445/) (2022)
- [DICE: Leveraging Sparsification for Out-of-Distribution Detection](https://mlsanity.com/paper/2111.09805/) (2021)
- [Unsupervised Domain Clusters in Pretrained Language Models](https://mlsanity.com/paper/2004.02105/) (2020)

## Built on by (most cited first, 29 of 29)

- [ARGS: Alignment as Reward-Guided Search](https://mlsanity.com/paper/2402.01694/) (2024)
- [Process Reward Model with Q-Value Rankings](https://mlsanity.com/paper/2410.11287/) (2024)
- [Dream the Impossible: Outlier Imagination with Diffusion Models](https://mlsanity.com/paper/2309.13415/) (2023)
- [On Large Language Model Continual Unlearning](https://mlsanity.com/paper/2407.10223/) (2024)
- [DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution](https://mlsanity.com/paper/2405.17357/) (2024)
- [How Does Unlabeled Data Provably Help Out-of-Distribution Detection?](https://mlsanity.com/paper/2402.03502/) (2024)
- [Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity](https://mlsanity.com/paper/2405.13967/) (2024)
- [Multi-Task Model Merging via Adaptive Weight Disentanglement](https://mlsanity.com/paper/2411.18729/) (2024)
- [How Good Are LLMs at Out-of-Distribution Detection?](https://mlsanity.com/paper/2308.10261/) (2023)
- [Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detector](https://mlsanity.com/paper/2404.08679/) (2024)
- [On the Learnability of Out-of-distribution Detection](https://mlsanity.com/paper/2404.04865/) (2024)
- [Shh, don't say that! Domain Certification in LLMs](https://mlsanity.com/paper/2502.19320/) (2025)
- [When and How Does In-Distribution Label Help Out-of-Distribution Detection?](https://mlsanity.com/paper/2405.18635/) (2024)
- [How Useful is Continued Pre-Training for Generative Unsupervised Domain Adaptation?](https://mlsanity.com/paper/2401.17514/) (2024)
- [Can Pre-trained Networks Detect Familiar Out-of-Distribution Data?](https://mlsanity.com/paper/2310.00847/) (2023)
- [Label Critic: Design Data Before Models](https://mlsanity.com/paper/2411.02753/) (2024)
- [RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training](https://mlsanity.com/paper/2312.04032/) (2023)
- [Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identification](https://mlsanity.com/paper/2503.01896/) (2025)
- [Trust The Typical](https://mlsanity.com/paper/2602.04581/) (2026)
- [Detecting out-of-distribution text using topological features of transformer-based language models](https://mlsanity.com/paper/2311.13102/) (2023)
- [A thorough benchmark of automatic text classification: From traditional approaches to large language models](https://mlsanity.com/paper/2504.01930/) (2025)
- [Medix: Out-of-Distribution Detection from Unlabeled Wild Data via Robust Gradient Statistics](https://mlsanity.com/paper/2510.06505/) (2025)
- [Foundation Model's Embedded Representations May Detect Distribution Shift](https://mlsanity.com/paper/2310.13836/) (2023)
- [VI-OOD: A Unified Representation Learning Framework for Textual Out-of-distribution Detection](https://mlsanity.com/paper/2404.06217/) (2024)
- [Diversity-grounded Channel Prototypical Learning for Out-of-Distribution Intent Detection](https://mlsanity.com/paper/2409.11114/) (2024)
- [Class-Aware Contrastive Optimization for Imbalanced Text Classification](https://mlsanity.com/paper/2410.22197/) (2024)
- [CTDGSI: A comprehensive exploitation of instance selection methods for automatic text classification. VII Concurso de Teses, Dissertacc~oes e Trabalhos de Graduacc~ao em SI -- XXI Simp'osio Brasileiro de Sistemas de Informacc~ao](https://mlsanity.com/paper/2506.07169/) (2025)
- [Domain Restriction via Multi SAE Layer Transitions](https://mlsanity.com/paper/2605.11920/) (2026)
- [Toward Robust In-Context Learning: Leveraging Out-of-distribution Proxies for Target Inaccessible Demonstration Retrieval](https://mlsanity.com/paper/2606.00014/) (2026)

## Abstract (arXiv)

Out-of-distribution (OOD) detection is a critical task for reliable predictions over text. Fine-tuning with pre-trained language models has been a de facto procedure to derive OOD detectors with respect to in-distribution (ID) data. Despite its common use, the understanding of the role of fine-tuning and its necessity for OOD detection is largely unexplored. In this paper, we raise the question: is fine-tuning necessary for OOD detection? We present a study investigating the efficacy of directly leveraging pre-trained language models for OOD detection, without any model fine-tuning on the ID data. We compare the approach with several competitive fine-tuning objectives, and offer new insights under various types of distributional shifts. Extensive evaluations on 8 diverse ID-OOD dataset pairs demonstrate near-perfect OOD detection performance (with 0% FPR95 in many cases), strongly outperforming its fine-tuned counterparts. We show that using distance-based detection methods, pre-trained language models are near-perfect OOD detectors when the distribution shift involves a domain change. Furthermore, we study the effect of fine-tuning on OOD detection and identify how to balance ID accuracy with OOD detection performance. Our code is publically available at https://github.com/Uppaal/lm-ood.