Understanding the Role of Self-Supervised Learning in Out-of-Distribution Detection Task
Not on the map. Why?
Abstract (arXiv)
Self-supervised learning (SSL) has achieved great success in a variety of computer vision tasks. However, the mechanism of how SSL works in these tasks remains a mystery. In this paper, we study how SSL can enhance the performance of the out-of-distribution (OOD) detection task. We first point out two general properties that a good OOD detector should have: 1) the overall feature space should be large and 2) the inlier feature space should be small. Then we demonstrate that SSL can indeed increase the intrinsic dimension of the overall feature space. In the meantime, SSL even has the potential to shrink the inlier feature space. As a result, there will be more space spared for the outliers, making OOD detection much easier. The conditions when SSL can shrink the inlier feature space is also discussed and validated. By understanding the role of SSL in the OOD detection task, our study can provide a guideline for designing better OOD detection algorithms. Moreover, this work can also shed light to other tasks where SSL can improve the performance.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 26 Oct 2021.
Builds on
29 of 38 references.
References · 29
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2016Wide Residual Networkscited by 3,057
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
- 2013Describing Textures in the Wildcited by 2,110
17 more
- 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
- 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
- 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
- 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
- 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
- 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
- 2018Deep Anomaly Detection with Outlier Exposurecited by 939
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
- 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
- 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
- 2021SSD: A Unified Framework for Self-Supervised Outlier Detectioncited by 264
- 2016Learning Representations for Automatic Colorizationcited by 251
- 2019Revisiting Self-Supervised Visual Representation Learningcited by 250
- 2018Anomaly Detection using One-Class Neural Networkscited by 82
- 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
- 2019Self-supervised Label Augmentation via Input Transformationscited by 15
- 2019Exploring Self-Supervised Regularization for Supervised and Semi-Supervised Learningcited by 4
Built on by
Citing papers per year: 2022: 1; 2023: 1; 2024: 1.
3 papers.