mlsanityData to 14 Oct 2026

Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detection

arXiv 2305.13282 ↗First posted 22 May 2023cs.CL cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

Out-of-distribution (OOD) detection is a critical task for reliable predictions over text. Fine-tuning with pre-trained language models has been a de facto procedure to derive OOD detectors with respect to in-distribution (ID) data. Despite its common use, the understanding of the role of fine-tuning and its necessity for OOD detection is largely unexplored. In this paper, we raise the question: is fine-tuning necessary for OOD detection? We present a study investigating the efficacy of directly leveraging pre-trained language models for OOD detection, without any model fine-tuning on the ID data. We compare the approach with several competitive fine-tuning objectives, and offer new insights under various types of distributional shifts. Extensive evaluations on 8 diverse ID-OOD dataset pairs demonstrate near-perfect OOD detection performance (with 0% FPR95 in many cases), strongly outperforming its fine-tuned counterparts. We show that using distance-based detection methods, pre-trained language models are near-perfect OOD detectors when the distribution shift involves a domain change. Furthermore, we study the effect of fine-tuning on OOD detection and identify how to balance ID accuracy with OOD detection performance. Our code is publically available at https://github.com/Uppaal/lm-ood.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 49 of its 61 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

49 of 61 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2020Supervised Contrastive Learningcited by 2,829
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  9. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  11. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  12. 2019Cross-lingual Language Model Pretrainingcited by 1,090
37 more
  1. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  2. 2017Selective Classification for Deep Neural Networkscited by 852
  3. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
  4. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  5. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 600
  6. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  7. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 453
  8. 2020Understanding the Behaviour of Contrastive Losscited by 398
  9. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
  10. 2020Calibration of Pre-trained Transformerscited by 315
  11. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 294
  12. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 276
  13. 2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 252
  14. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  15. 2020Selective Question Answering under Domain Shiftcited by 238
  16. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 216
  17. 2021DICE: Leveraging Sparsification for Out-of-Distribution Detectioncited by 166
  18. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  19. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 144
  20. 2022How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?cited by 128
  21. 2021Types of Out-of-Distribution Texts and How to Detect Themcited by 80
  22. 2021The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuningcited by 75
  23. 2021Contrastive Out-of-Distribution Detection for Pretrained Transformerscited by 71
  24. 2021Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detectioncited by 70
  25. 2020Trust Issues: Uncertainty Estimation Does Not Enable Reliable OOD Detection On Medical Tabular Datacited by 52
  26. 2022News Category Datasetcited by 43
  27. 2021Modeling Discriminative Representations for Out-of-Domain Detection with Supervised Contrastive Learningcited by 42
  28. 2021Out-of-Scope Intent Detection with Self-Supervision and Discriminative Trainingcited by 42
  29. 2019Out-of-Domain Detection for Low-Resource Text Classification Taskscited by 36
  30. 2021Unsupervised Out-of-Domain Detection via Pre-trained Transformerscited by 31
  31. 2017Distance-based Confidence Score for Neural Network Classifierscited by 30
  32. 2021Enhancing the Generalization for Intent Classification and Out-of-Domain Detection in SLUcited by 22
  33. 2021GOLD: Improving Out-of-Scope Detection in Dialogues using Data Augmentationcited by 20
  34. 2022Towards Textual Out-of-Domain Detection without In-Domain Labelscited by 14
  35. 2021PnPOOD : Out-Of-Distribution Detection for Text Classification via Plug andPlay Data Augmentationcited by 12
  36. 2021kFolden: k-Fold Ensemble for Out-Of-Distribution Detectioncited by 10
  37. 2021ProtoInfoMax: Prototypical Networks with Mutual Information Maximization for Out-of-Domain Detectioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 15; 2025: 5; 2026: 3.

29 papers.

Sort:

By area:

  1. 2024ARGS: Alignment as Reward-Guided Searchcited by 103
  2. 2024Process Reward Model with Q-Value Rankingscited by 80
  3. 2023Dream the Impossible: Outlier Imagination with Diffusion Modelscited by 77
  4. 2024On Large Language Model Continual Unlearningcited by 52
  5. 2024DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distributioncited by 43
  6. 2024How Does Unlabeled Data Provably Help Out-of-Distribution Detection?cited by 36
  7. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicitycited by 35
  8. 2024Multi-Task Model Merging via Adaptive Weight Disentanglementcited by 22
  9. 2023How Good Are LLMs at Out-of-Distribution Detection?cited by 17
  10. 2024Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detectorcited by 11
  11. 2024On the Learnability of Out-of-distribution Detectioncited by 8
  12. 2025Shh, don't say that! Domain Certification in LLMscited by 7
17 more
  1. 2024When and How Does In-Distribution Label Help Out-of-Distribution Detection?cited by 6
  2. 2024How Useful is Continued Pre-Training for Generative Unsupervised Domain Adaptation?cited by 5
  3. 2023Can Pre-trained Networks Detect Familiar Out-of-Distribution Data?cited by 4
  4. 2024Label Critic: Design Data Before Modelscited by 4
  5. 2023RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Trainingcited by 3
  6. 2025Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identificationcited by 3
  7. 2026Trust The Typicalcited by 3
  8. 2023Detecting out-of-distribution text using topological features of transformer-based language modelscited by 2
  9. 2025A thorough benchmark of automatic text classification: From traditional approaches to large language modelscited by 2
  10. 2025Medix: Out-of-Distribution Detection from Unlabeled Wild Data via Robust Gradient Statisticscited by 2
  11. 2023Foundation Model's Embedded Representations May Detect Distribution Shiftcited by 1
  12. 2024VI-OOD: A Unified Representation Learning Framework for Textual Out-of-distribution Detectioncited by 0
  13. 2024Diversity-grounded Channel Prototypical Learning for Out-of-Distribution Intent Detectioncited by 0
  14. 2024Class-Aware Contrastive Optimization for Imbalanced Text Classificationcited by 0
  15. 2025CTDGSI: A comprehensive exploitation of instance selection methods for automatic text classification. VII Concurso de Teses, Dissertacc~oes e Trabalhos de Graduacc~ao em SI -- XXI Simp'osio Brasileiro de Sistemas de Informacc~aocited by 0
  16. 2026Domain Restriction via Multi SAE Layer Transitionscited by 0
  17. 2026Toward Robust In-Context Learning: Leveraging Out-of-distribution Proxies for Target Inaccessible Demonstration Retrievalcited by 0