# Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Need

arXiv: https://arxiv.org/abs/2303.15256 · HTML: https://mlsanity.com/paper/2303.15256/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 57 references that are in the map.
- First posted on arXiv 27 Mar 2023.

## Builds on (most cited first, 30 of 30)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Revisiting Semi-Supervised Learning with Graph Embeddings](https://mlsanity.com/paper/1603.08861/) (2016)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [S4L: Self-Supervised Semi-Supervised Learning](https://mlsanity.com/paper/1905.03670/) (2019)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced](https://mlsanity.com/paper/1806.00900/) (2018)
- [Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods](https://mlsanity.com/paper/2205.11508/) (2022)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [Towards artificial general intelligence via a multimodal foundation model](https://mlsanity.com/paper/2110.14378/) (2021)
- [Sequential Experimental Design for Transductive Linear Bandits](https://mlsanity.com/paper/1906.08399/) (2019)
- [Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels](https://mlsanity.com/paper/2103.13646/) (2021)
- [Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorization](https://mlsanity.com/paper/2106.14289/) (2021)
- [Machine Teaching: A New Paradigm for Building Machine Learning Systems](https://mlsanity.com/paper/1707.06742/) (2017)
- [The SSL Interplay: Augmentations, Inductive Bias, and Generalization](https://mlsanity.com/paper/2302.02774/) (2023)
- [Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorization](https://mlsanity.com/paper/2203.02839/) (2022)
- [Sharp Analysis of Learning with Discrete Losses](https://mlsanity.com/paper/1810.06839/) (2018)
- [Joint Embedding Self-Supervised Learning in the Kernel Regime](https://mlsanity.com/paper/2209.14884/) (2022)
- [MaxiMin Active Learning in Overparameterized Model Classes}](https://mlsanity.com/paper/1905.12782/) (2019)
- [On minimal variations for unsupervised representation learning](https://mlsanity.com/paper/2211.03782/) (2022)
- [Active Labeling: Streaming Stochastic Gradients](https://mlsanity.com/paper/2205.13255/) (2022)

## Built on by (most cited first, 9 of 9)

- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [mathbbX-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs](https://mlsanity.com/paper/2407.18134/) (2024)
- [Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?](https://mlsanity.com/paper/2406.10743/) (2024)
- [Rethinking Positive Pairs in Contrastive Learning](https://mlsanity.com/paper/2410.18200/) (2024)
- [ActiveDC: Distribution Calibration for Active Finetuning](https://mlsanity.com/paper/2311.07634/) (2023)
- [GPS-SSL: Guided Positive Sampling to Inject Prior Into Self-Supervised Learning](https://mlsanity.com/paper/2401.01990/) (2024)
- [Optimal and Efficient Binary Questioning for Human-in-the-Loop Annotation](https://mlsanity.com/paper/2307.01578/) (2023)
- [MyriadAL: Active Few Shot Learning for Histopathology](https://mlsanity.com/paper/2310.16161/) (2023)
- [ADROIT: A Self-Supervised Framework for Learning Robust Representations for Active Learning](https://mlsanity.com/paper/2503.07506/) (2025)

## Abstract (arXiv)

Self-Supervised Learning (SSL) has emerged as the solution of choice to learn transferable representations from unlabeled data. However, SSL requires to build samples that are known to be semantically akin, i.e. positive views. Requiring such knowledge is the main limitation of SSL and is often tackled by ad-hoc strategies e.g. applying known data-augmentations to the same input. In this work, we formalize and generalize this principle through Positive Active Learning (PAL) where an oracle queries semantic relationships between samples. PAL achieves three main objectives. First, it unveils a theoretically grounded learning framework beyond SSL, based on similarity graphs, that can be extended to tackle supervised and semi-supervised learning depending on the employed oracle. Second, it provides a consistent algorithm to embed a priori knowledge, e.g. some observed labels, into any SSL losses without any change in the training pipeline. Third, it provides a proper active learning framework yielding low-cost solutions to annotate datasets, arguably bringing the gap between theory and practice of active learning that is based on simple-to-answer-by-non-experts queries of semantic relationships between inputs.