mlsanityData to 14 Oct 2026

Dataset-Level Regularization for Joint-Embedding Self-Supervised Learning

arXiv 2610.20158 ↗First posted 14 Oct 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Self-supervised joint embedding learning aligns representations of augmented views while regularizing their distribution to prevent collapse. Regularization aims to shape the overall representation distribution, but most joint embedding methods restrict its computation to the active minibatch. We propose out-of-batch regularization, which maintains a dataset-level state in CPU memory and uses globally constructed per-instance targets during minibatch training. Evaluated on natural images and biomedical datasets, these anchors approximate the full-dataset regularization gradient more faithfully than conventional minibatch estimation at small batch sizes. Applications to contrastive (SimCLR) and non-contrastive (SIGReg) approaches show consistent improvement over the corresponding in-batch objectives for image classification, minority-class performance under controlled imbalance, instance retrieval, and continued self-supervised pretraining, with the largest gains at small batch sizes. These results suggest that dataset-level regularization can be decoupled from minibatch optimization, making batch size an optimization choice that does not affect the geometry of the regularizer.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 47 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

39 of 47 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
27 more
  1. 2013Describing Textures in the Wildcited by 2,125
  2. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  4. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  5. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  6. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  7. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  8. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 690
  9. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  10. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 422
  11. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  12. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  13. 2019On Mutual Information Maximization for Representation Learningcited by 316
  14. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  15. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  16. 2021Decoupled Contrastive Learningcited by 143
  17. 2019Cross-Batch Memory for Embedding Learningcited by 132
  18. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 127
  19. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 102
  20. 2017Unsupervised Learning by Predicting Noisecited by 98
  21. 2021Self-Supervised Pretraining Improves Self-Supervised Pretrainingcited by 82
  22. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 58
  23. 2022Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performancecited by 43
  24. 2023Unsupervised Learning on a DIET: Datum IndEx as Target Free of Self-Supervision, Reconstruction, Projector Headcited by 4
  25. 2025DIET-CP: Lightweight and Data Efficient Self Supervised Continued Pretrainingcited by 3
  26. 2024EMC2: Efficient MCMC Negative Sampling for Contrastive Learning with Global Convergencecited by 2
  27. 2026IConE: Batch Independent Collapse Prevention for Self-Supervised Representation Learningcited by 1

Built on by

Not cited yet.