mlsanityData to 7 Oct 2026

Blockwise Self-Supervised Learning at Scale

arXiv 2302.01647 ↗First posted 3 Feb 2023cs.CV cs.AI cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Current state-of-the-art deep networks are all powered by backpropagation. In this paper, we explore alternatives to full backpropagation in the form of blockwise learning rules, leveraging the latest developments in self-supervised learning. We show that a blockwise pretraining procedure consisting of training independently the 4 main blocks of layers of a ResNet-50 with Barlow Twins' loss function at each block performs almost as well as end-to-end backpropagation on ImageNet: a linear probe trained on top of our blockwise pretrained model obtains a top-1 classification accuracy of 70.48%, only 1.1% below the accuracy of an end-to-end pretrained network (71.57% accuracy). We perform extensive experiments to understand the impact of different components within our method and explore a variety of adaptations of self-supervised learning to the blockwise paradigm, building an exhaustive understanding of the critical avenues for scaling local learning rules to large networks, with implications ranging from hardware design to neuroscience.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 3 Feb 2023.

Builds on

28 of 38 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  12. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
16 more
  1. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  2. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  3. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  4. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  5. 2014Difference Target Propagationcited by 144
  6. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  7. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 113
  8. 2019Putting An End to End-to-End: Gradient-Isolated Learning of Representationscited by 85
  9. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  10. 2022Hebbian Deep Learning Without Feedbackcited by 61
  11. 2020LoCo: Local Contrastive Representation Learningcited by 55
  12. 2020Local plasticity rules can learn deep representations using self-supervised contrastive predictionscited by 47
  13. 2022Towards Scaling Difference Target Propagation by Learning Backprop Targetscited by 34
  14. 2021Credit Assignment Through Broadcasting a Global Error Vectorcited by 19
  15. 2020GAIT-prop: A biologically plausible learning rule derived from backpropagation of errorcited by 15
  16. 2021Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networkscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 7; 2025: 4; 2026: 5.

22 papers.

Sort:

  1. 2024Training of Physical Neural Networkscited by 47
  2. 2023Backpropagation-free Training of Deep Physical Neural Networkscited by 42
  3. 2023Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignmentcited by 13
  4. 2025DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretationcited by 9
  5. 2024End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Trainingcited by 8
  6. 2025NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagationcited by 8
  7. 2023Block-local learning with probabilistic latent representationscited by 6
  8. 2023A Graph is Worth 1-bit Spikes: When Graph Contrastive Learning Meets Spiking Neural Networkscited by 6
  9. 2024MLAAN: Scaling Supervised Local Learning with Multilaminar Leap Augmented Auxiliary Networkcited by 5
  10. 2024HPFF: Hierarchical Locally Supervised Learning with Patch Feature Fusioncited by 5
  11. 2024Towards Interpretable Deep Local Learning with Successive Gradient Reconciliationcited by 4
  12. 2023Layerwise complexity-matched learning yields an improved model of cortical area V2cited by 1
10 more
  1. 2024Unsupervised End-to-End Training with a Self-Defined Targetcited by 1
  2. 2025DeInfoReg: A Decoupled Learning Framework for Better Training Throughputcited by 1
  3. 2023Oversmoothing: A Nightmare for Graph Contrastive Learning?cited by 0
  4. 2024Masked Image Modeling as a Framework for Self-Supervised Learning across Eye Movementscited by 0
  5. 2025Precoder Design in Multi-User FDD Systems with VQ-VAE and GNNcited by 0
  6. 2026Depth-Wise Representation Development Under Blockwise Self-Supervised Learning for Video Vision Transformerscited by 0
  7. 2026Can Local Learning Match Self-Supervised Backpropagation?cited by 0
  8. 2026LocalProp: Neuro-Localized Memory-Efficient Backpropagationcited by 0
  9. 2026SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learningcited by 0
  10. 2026Increasing Width Allows Greedy Layer-wise Training to Rival End-to-End Backpropagation in Self-Supervised Learningcited by 0