mlsanityData to 7 Oct 2026

Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Need

arXiv 2303.15256 ↗First posted 27 Mar 2023cs.LG cs.AI cs.HCcited by 9

Not on the map. Why?

Abstract (arXiv)

Self-Supervised Learning (SSL) has emerged as the solution of choice to learn transferable representations from unlabeled data. However, SSL requires to build samples that are known to be semantically akin, i.e. positive views. Requiring such knowledge is the main limitation of SSL and is often tackled by ad-hoc strategies e.g. applying known data-augmentations to the same input. In this work, we formalize and generalize this principle through Positive Active Learning (PAL) where an oracle queries semantic relationships between samples. PAL achieves three main objectives. First, it unveils a theoretically grounded learning framework beyond SSL, based on similarity graphs, that can be extended to tackle supervised and semi-supervised learning depending on the employed oracle. Second, it provides a consistent algorithm to embed a priori knowledge, e.g. some observed labels, into any SSL losses without any change in the training pipeline. Third, it provides a proper active learning framework yielding low-cost solutions to annotate datasets, arguably bringing the gap between theory and practice of active learning that is based on simple-to-answer-by-non-experts queries of semantic relationships between inputs.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 30 of its 57 references that are in the map. First posted on arXiv 27 Mar 2023.

Builds on

30 of 57 references.

References · 30

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  5. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  6. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  7. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 970
  8. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  9. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  10. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  11. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 370
  12. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
18 more
  1. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  2. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  3. 2021Curriculum Learning: A Surveycited by 257
  4. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  5. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  6. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  7. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  8. 2019Sequential Experimental Design for Transductive Linear Banditscited by 88
  9. 2021Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labelscited by 69
  10. 2021Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorizationcited by 44
  11. 2017Machine Teaching: A New Paradigm for Building Machine Learning Systemscited by 41
  12. 2023The SSL Interplay: Augmentations, Inductive Bias, and Generalizationcited by 40
  13. 2022Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorizationcited by 34
  14. 2018Sharp Analysis of Learning with Discrete Lossescited by 13
  15. 2022Joint Embedding Self-Supervised Learning in the Kernel Regimecited by 12
  16. 2019MaxiMin Active Learning in Overparameterized Model Classes}cited by 10
  17. 2022On minimal variations for unsupervised representation learningcited by 6
  18. 2022Active Labeling: Streaming Stochastic Gradientscited by 3

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 4; 2025: 1.

9 papers.

  1. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  2. 2024mathbbX-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphscited by 9
  3. 2024Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?cited by 8
  4. 2024Rethinking Positive Pairs in Contrastive Learningcited by 5
  5. 2023ActiveDC: Distribution Calibration for Active Finetuningcited by 3
  6. 2024GPS-SSL: Guided Positive Sampling to Inject Prior Into Self-Supervised Learningcited by 3
  7. 2023Optimal and Efficient Binary Questioning for Human-in-the-Loop Annotationcited by 0
  8. 2023MyriadAL: Active Few Shot Learning for Histopathologycited by 0
  9. 2025ADROIT: A Self-Supervised Framework for Learning Robust Representations for Active Learningcited by 0