mlsanityData to 14 Oct 2026

Barlow Twins: Self-Supervised Learning via Redundancy Reduction

arXiv 2103.03230 ↗First posted 4 Mar 2021cs.CV cs.AI cs.LGcited by 1,838

Not on the map. Why?

Cited by 48 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks. A successful approach to SSL is to learn embeddings which are invariant to distortions of the input sample. However, a recurring issue with this approach is the existence of trivial constant solutions. Most current methods avoid such solutions by careful implementation details. We propose an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample, and making it as close to the identity matrix as possible. This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors. The method is called Barlow Twins, owing to neuroscientist H. Barlow's redundancy-reduction principle applied to a pair of identical networks. Barlow Twins does not require large batches nor asymmetry between the network twins such as a predictor network, gradient stopping, or a moving average on the weight updates. Intriguingly it benefits from very high-dimensional output vectors. Barlow Twins outperforms previous methods on ImageNet for semi-supervised classification in the low-data regime, and is on par with current state of the art for ImageNet classification with a linear classifier head, and for transfer tasks of classification and object detection.

Key facts. Cited by 1,838 papers in the map (data through 14 Oct 2026). Builds on 30 of its 52 references that are in the map. First posted on arXiv 4 Mar 2021.

Builds on

30 of 52 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2017Mask R-CNNcited by 7,277
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
18 more
  1. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  2. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  3. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  4. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  5. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  6. 2016End-to-end Optimized Image Compressioncited by 696
  7. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  8. 2017Large Batch Training of Convolutional Networkscited by 535
  9. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  10. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  11. 2020Whitening for Self-Supervised Representation Learningcited by 279
  12. 2020Contrastive Clusteringcited by 265
  13. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  14. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  15. 2017Unsupervised Learning by Predicting Noisecited by 98
  16. 2020BYOL works even without batch statisticscited by 95
  17. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  18. 2019A Unified Theory of Early Visual Representations from Retina to Cortex through Anatomically Constrained Deep CNNscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 161; 2022: 420; 2023: 405; 2024: 353; 2025: 251; 2026: 248.

1,838 papers.

Sort:

By area:

  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  2. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  3. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  4. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  5. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 363
  6. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  7. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  8. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  9. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  10. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  11. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  12. 2022Contrastive Test-Time Adaptationcited by 261
38 more
  1. 2023A Cookbook of Self-Supervised Learningcited by 250
  2. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  3. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  4. 2022A Self-Supervised Descriptor for Image Copy Detectioncited by 176
  5. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  6. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 169
  7. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 164
  8. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 161
  9. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  10. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 156
  11. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  12. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  13. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 150
  14. 2021Self-Supervised Models are Continual Learnerscited by 149
  15. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  16. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 148
  17. 20213D Infomax improves GNNs for Molecular Property Predictioncited by 146
  18. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  19. 2021Decoupled Contrastive Learningcited by 143
  20. 2021From Canonical Correlation Analysis to Self-supervised Graph Neural Networkscited by 141
  21. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  22. 2021Self-supervised Learning on Graphs: Contrastive, Generative,or Predictivecited by 136
  23. 2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
  24. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  25. 2022Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detectioncited by 130
  26. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  27. 2021Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bankcited by 125
  28. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 124
  29. 2023Change is Hard: A Closer Look at Subpopulation Shiftcited by 120
  30. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  31. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  32. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  33. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  34. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  35. 2021An Empirical Study of Graph Contrastive Learningcited by 113
  36. 2021Aligning Pretraining for Detection via Object-Level Contrastive Learningcited by 110
  37. 2021Efficient Training of Visual Transformers with Small Datasetscited by 108
  38. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108

50 most cited of 1,838 All 1,838 (JSON)