# On the Effectiveness of Out-of-Distribution Data in Self-Supervised Long-Tail Learning

arXiv: https://arxiv.org/abs/2306.04934 · HTML: https://mlsanity.com/paper/2306.04934/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 53 references that are in the map.
- First posted on arXiv 8 Jun 2023.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [Rethinking the Value of Labels for Improving Class-Imbalanced Learning](https://mlsanity.com/paper/2006.07529/) (2020)
- [What Do Compressed Deep Neural Networks Forget?](https://mlsanity.com/paper/1911.05248/) (2019)
- [Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective](https://mlsanity.com/paper/2003.10780/) (2020)
- [Targeted Supervised Contrastive Learning for Long-Tailed Recognition](https://mlsanity.com/paper/2111.13998/) (2021)
- [Self-supervised Learning is More Robust to Dataset Imbalance](https://mlsanity.com/paper/2110.05025/) (2021)
- [Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning](https://mlsanity.com/paper/2007.08844/) (2020)
- [Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap](https://mlsanity.com/paper/2203.13457/) (2022)
- [Deep Active Learning over the Long Tail](https://mlsanity.com/paper/1711.00941/) (2017)
- [Open Compound Domain Adaptation](https://mlsanity.com/paper/1909.03403/) (2019)
- [Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation](https://mlsanity.com/paper/2010.04647/) (2020)

## Built on by (most cited first, 20 of 20)

- [CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No](https://mlsanity.com/paper/2308.12213/) (2023)
- [Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation](https://mlsanity.com/paper/2501.06693/) (2025)
- [Out-of-Distribution Detection in Long-Tailed Recognition with Calibrated Outlier Class Learning](https://mlsanity.com/paper/2312.10686/) (2023)
- [ConjNorm: Tractable Density Estimation for Out-of-Distribution Detection](https://mlsanity.com/paper/2402.17888/) (2024)
- [Towards Distribution-Agnostic Generalized Category Discovery](https://mlsanity.com/paper/2310.01376/) (2023)
- [A Square Peg in a Square Hole: Meta-Expert for Long-Tailed Semi-Supervised Learning](https://mlsanity.com/paper/2505.16341/) (2025)
- [Towards Heterogeneous Long-tailed Learning: Benchmarking, Metrics, and Toolbox](https://mlsanity.com/paper/2307.08235/) (2023)
- [Frequency-Aware Self-Supervised Long-Tailed Learning](https://mlsanity.com/paper/2309.04723/) (2023)
- [Long-Tailed Out-of-Distribution Detection via Normalized Outlier Distribution Adaptation](https://mlsanity.com/paper/2410.20807/) (2024)
- [Passive Deepfake Detection Across Multi-modalities: A Comprehensive Survey](https://mlsanity.com/paper/2411.17911/) (2024)
- [Making Reliable and Flexible Decisions in Long-tailed Classification](https://mlsanity.com/paper/2501.14090/) (2025)
- [Gradient Short-Circuit: Efficient Out-of-Distribution Detection via Feature Intervention](https://mlsanity.com/paper/2507.01417/) (2025)
- [Divide, Weight, and Route: Difficulty-Aware Optimization with Dynamic Expert Fusion for Long-tailed Recognition](https://mlsanity.com/paper/2508.19630/) (2025)
- [Maximally Useful and Minimally Redundant: The Key to Self Supervised Learning for Imbalanced Data](https://mlsanity.com/paper/2509.08469/) (2025)
- [A Self supervised learning framework for imbalanced medical imaging datasets](https://mlsanity.com/paper/2604.01947/) (2026)
- [PAT: Pixel-wise Adaptive Training for Long-tailed Segmentation](https://mlsanity.com/paper/2404.05393/) (2024)
- [Rethinking Open-World Semi-Supervised Learning: Distribution Mismatch and Inductive Inference](https://mlsanity.com/paper/2405.20829/) (2024)
- [On the Generalization and Causal Explanation in Self-Supervised Learning](https://mlsanity.com/paper/2410.00772/) (2024)
- [RICASSO: Reinforced Imbalance Learning with Class-Aware Self-Supervised Outliers Exposure](https://mlsanity.com/paper/2410.10548/) (2024)
- [Unsupervised Contrastive Learning Using Out-Of-Distribution Data for Long-Tailed Dataset](https://mlsanity.com/paper/2506.12698/) (2025)

## Abstract (arXiv)

Though Self-supervised learning (SSL) has been widely studied as a promising technique for representation learning, it doesn't generalize well on long-tailed datasets due to the majority classes dominating the feature space. Recent work shows that the long-tailed learning performance could be boosted by sampling extra in-domain (ID) data for self-supervised training, however, large-scale ID data which can rebalance the minority classes are expensive to collect. In this paper, we propose an alternative but easy-to-use and effective solution, Contrastive with Out-of-distribution (OOD) data for Long-Tail learning (COLT), which can effectively exploit OOD data to dynamically re-balance the feature space. We empirically identify the counter-intuitive usefulness of OOD samples in SSL long-tailed learning and principally design a novel SSL method. Concretely, we first localize the `head' and `tail' samples by assigning a tailness score to each OOD sample based on its neighborhoods in the feature space. Then, we propose an online OOD sampling strategy to dynamically re-balance the feature space. Finally, we enforce the model to be capable of distinguishing ID and OOD samples by a distribution-level supervised contrastive loss. Extensive experiments are conducted on various datasets and several state-of-the-art SSL frameworks to verify the effectiveness of the proposed method. The results show that our method significantly improves the performance of SSL on long-tailed datasets by a large margin, and even outperforms previous work which uses external ID data. Our code is available at https://github.com/JianhongBai/COLT.