mlsanityData to 7 Oct 2026

A Cookbook of Self-Supervised Learning

arXiv 2304.12210 ↗First posted 24 Apr 2023cs.LG cs.CVcited by 248

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning. Yet, much like cooking, training SSL methods is a delicate art with a high barrier to entry. While many components are familiar, successfully training a SSL method involves a dizzying set of choices from the pretext tasks to training hyper-parameters. Our goal is to lower the barrier to entry into SSL research by laying the foundations and latest SSL recipes in the style of a cookbook. We hope to empower the curious researcher to navigate the terrain of methods, understand the role of the various knobs, and gain the know-how required to explore how delicious SSL can be.

Key facts. Cited by 248 papers in the map (data through 7 Oct 2026). Builds on 252 of its 313 references that are in the map. First posted on arXiv 24 Apr 2023.

Builds on

252 of 313 references.

References · 252

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2017Mask R-CNNcited by 7,259
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2013Auto-Encoding Variational Bayescited by 3,980
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  13. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  14. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  15. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  16. 2016Improved Techniques for Training GANscited by 2,938
  17. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  18. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  19. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  20. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  21. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  22. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  23. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  24. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  25. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  26. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  27. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  28. 2020Reformer: The Efficient Transformercited by 1,658
  29. 2021ViViT: A Video Vision Transformercited by 1,610
  30. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  31. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  32. 2020Rethinking Attention with Performerscited by 1,584
  33. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  34. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  35. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  36. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  37. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  38. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393

50 most cited of 252 All 252 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 70; 2024: 95; 2025: 60; 2026: 23.

248 papers.

Sort:

By area:

  1. 2023Simple and Controllable Music Generationcited by 480
  2. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  3. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 199
  4. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 144
  5. 2024Measuring Style Similarity in Diffusion Modelscited by 106
  6. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 97
  7. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  8. 2024MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learningcited by 76
  9. 2025Scaling Language-Free Visual Representation Learningcited by 62
  10. 2024Learning by Reconstruction Produces Uninformative Features For Perceptioncited by 54
  11. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 53
  12. 2024Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biologycited by 52
38 more
  1. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustnesscited by 48
  2. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  3. 2024Training of Physical Neural Networkscited by 47
  4. 2023Three Pillars improving Vision Foundation Model Distillation for Lidarcited by 43
  5. 2023A General-Purpose Self-Supervised Model for Computational Pathologycited by 42
  6. 2023SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-trainingcited by 41
  7. 2024What to align in multimodal contrastive learning?cited by 41
  8. 2023Visual cognition in multimodal large language modelscited by 38
  9. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  10. 2024Foundation Models for Music: A Surveycited by 34
  11. 2023Reverse Engineering Self-Supervised Learningcited by 33
  12. 2025Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learningcited by 33
  13. 2023M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Modelscited by 30
  14. 2024HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognitioncited by 30
  15. 2023Self-Supervised Learning with Lie Symmetries for Partial Differential Equationscited by 29
  16. 2023Machine Learning for Anomaly Detection in Particle Physicscited by 29
  17. 2025Can Masked Autoencoders Also Listen to Birds?cited by 26
  18. 2023Group Robust Classification Without Any Group Informationcited by 25
  19. 2024PhilEO Bench: Evaluating Geo-Spatial Foundation Modelscited by 25
  20. 2024SARATR-X: Toward Building A Foundation Model for SAR Target Recognitioncited by 25
  21. 2024BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversitycited by 22
  22. 2023CL-MAE: Curriculum-Learned Masked Autoencoderscited by 21
  23. 2024The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learningcited by 21
  24. 2023Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Modelcited by 19
  25. 2024A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classificationcited by 19
  26. 2023Mesogeos: A multi-purpose dataset for data-driven wildfire modeling in the Mediterraneancited by 18
  27. 2023Deep Generative Models for Detector Signature Simulation: A Taxonomic Reviewcited by 18
  28. 2024UniCorn: A Unified Contrastive Learning Approach for Multi-view Molecular Representation Learningcited by 18
  29. 2023ViP: A Differentially Private Foundation Model for Computer Visioncited by 17
  30. 2023Self-Supervised Pre-Training with Contrastive and Masked Autoencoder Methods for Dealing with Small Datasets in Deep Learning for Medical Imagingcited by 16
  31. 2024Efficient Image Pre-Training with Siamese Cropped Masked Autoencoderscited by 16
  32. 2024A Review of Pseudo-Labeling for Computer Visioncited by 16
  33. 2024A Survey on Self-Supervised Learning for Non-Sequential Tabular Datacited by 15
  34. 2024HeAR -- Health Acoustic Representationscited by 15
  35. 2023Self-supervised Representation Learning From Random Data Projectorscited by 14
  36. 2024SwiMDiff: Scene-wide Matching Contrastive Learning with Diffusion Constraint for Remote Sensing Imagecited by 14
  37. 2023Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenescited by 13
  38. 2023Adversarial Attacks on Foundational Vision Modelscited by 13

50 most cited of 248 All 248 (JSON)