Double Descent Meets Out-of-Distribution Detection: Theoretical Insights and Empirical Analysis on the role of model complexity
Not on the map. Why?
Abstract (arXiv)
Out-of-distribution (OOD) detection is essential for ensuring the reliability and safety of machine learning systems. In recent years, it has received increasing attention, particularly through post-hoc detection and training-based methods. In this paper, we focus on post-hoc OOD detection, which enables identifying OOD samples without altering the model's training procedure or objective. Our primary goal is to investigate the relationship between model capacity and its OOD detection performance. Specifically, we aim to answer the following question: Does the Double Descent phenomenon manifest in post-hoc OOD detection? This question is crucial, as it can reveal whether overparameterization, which is already known to benefit generalization, can also enhance OOD detection. Despite the growing interest in these topics by the classic supervised machine learning community, this intersection remains unexplored for OOD detection. We empirically demonstrate that the Double Descent effect does indeed appear in post-hoc OOD detection. Furthermore, we provide theoretical insights to explain why this phenomenon emerges in such setting. Finally, we show that the overparameterized regime does not yield superior results consistently, and we propose a method to identify the optimal regime for OOD detection based on our observations.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 59 of its 79 references that are in the map. First posted on arXiv 4 Nov 2024.
Builds on
59 of 79 references.
References · 59
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2013Describing Textures in the Wildcited by 2,110
- 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
- 2019Natural Adversarial Examplescited by 1,401
- 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
- 2020Energy-based Out-of-distribution Detectioncited by 1,296
38 more
- 2017Deeper, Broader and Artier Domain Generalizationcited by 983
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
- 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
- 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
- 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
- 2018Deep Visual Domain Adaptation: A Surveycited by 634
- 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
- 2017Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discoverycited by 566
- 2019Benign Overfitting in Linear Regressioncited by 544
- 2015Towards Open Set Deep Networkscited by 533
- 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
- 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
- 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
- 2022ViM: Out-Of-Distribution with Virtual-logit Matchingcited by 404
- 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
- 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
- 2018Learning Confidence for Out-of-Distribution Detection in Neural Networkscited by 245
- 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
- 2022Extremely Simple Activation Shaping for Out-of-Distribution Detectioncited by 212
- 2019Two models of double descent for weak featurescited by 194
- 2018To understand deep learning we need to understand kernel learningcited by 186
- 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
- 2022How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?cited by 128
- 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
- 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
- 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
- 2017A Random Matrix Approach to Neural Networkscited by 80
- 2016Places: An Image Database for Deep Scene Understandingcited by 76
- 2020A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descentcited by 69
- 2020Implicit Regularization of Random Feature Modelscited by 54
- 2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
- 2023NECO: NEural Collapse Based Out-of-distribution detectioncited by 41
- 2020Kernel regression in high dimensions: Refined analysis beyond double descentcited by 39
- 2019Exact expressions for double descent and implicit regularization via surrogate random designcited by 35
- 2020Analytic Study of Double Descent in Binary Classification: The Impact of Losscited by 28
- 2022Linking Neural Collapse and L2 Normalization with Improved Out-of-Distribution Detection in Deep Neural Networkscited by 23
- 2021When and how epochwise double descent happenscited by 20
50 most cited of 59 All 59 (JSON)
Built on by
Not cited yet.