# Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks

arXiv: https://arxiv.org/abs/2109.15089 · HTML: https://mlsanity.com/paper/2109.15089/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 56 references that are in the map.
- First posted on arXiv 30 Sep 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Understanding self-supervised Learning Dynamics without Contrastive Pairs](https://mlsanity.com/paper/2102.06810/) (2021)
- [Difference Target Propagation](https://mlsanity.com/paper/1412.7525/) (2014)
- [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016)
- [Training Neural Networks with Local Error Signals](https://mlsanity.com/paper/1901.06656/) (2019)
- [Greedy Layerwise Learning Can Scale to ImageNet](https://mlsanity.com/paper/1812.11446/) (2018)
- [Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures](https://mlsanity.com/paper/1807.04587/) (2018)
- [Deep Learning without Weight Transport](https://mlsanity.com/paper/1904.05391/) (2019)
- [Putting An End to End-to-End: Gradient-Isolated Learning of Representations](https://mlsanity.com/paper/1905.11786/) (2019)
- [How Important is Weight Symmetry in Backpropagation?](https://mlsanity.com/paper/1510.05067/) (2015)
- [A Theoretical Framework for Target Propagation](https://mlsanity.com/paper/2006.14331/) (2020)
- [Local plasticity rules can learn deep representations using self-supervised contrastive predictions](https://mlsanity.com/paper/2010.08262/) (2020)
- [Deep Learning Models of the Retinal Response to Natural Scenes](https://mlsanity.com/paper/1702.01825/) (2017)
- [Learning Deep ResNet Blocks Sequentially using Boosting Theory](https://mlsanity.com/paper/1706.04964/) (2017)
- [Deep learning with asymmetric connections and Hebbian updates](https://mlsanity.com/paper/1812.07965/) (2018)

## Built on by (most cited first, 6 of 6)

- [Blockwise Self-Supervised Learning at Scale](https://mlsanity.com/paper/2302.01647/) (2023)
- [Self-Contrastive Forward-Forward Algorithm](https://mlsanity.com/paper/2409.11593/) (2024)
- [NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagation](https://mlsanity.com/paper/2503.24322/) (2025)
- [Representation Learning in a Decomposed Encoder Design for Bio-inspired Hebbian Learning](https://mlsanity.com/paper/2401.08603/) (2023)
- [Closed-Form Feedback-Free Learning with Forward Projection](https://mlsanity.com/paper/2501.16476/) (2025)
- [Skewed Neuronal Heterogeneity Enhances Efficiency On Various Computing Systems](https://mlsanity.com/paper/2412.05126/) (2024)

## Abstract (arXiv)

We develop biologically plausible training mechanisms for self-supervised learning (SSL) in deep networks. Specifically, by biological plausible training we mean (i) All updates of weights are based on current activities of pre-synaptic units and current, or activity retrieved from short term memory of post synaptic units, including at the top-most error computing layer, (ii) Complex computations such as normalization, inner products and division are avoided (iii) Asymmetric connections between units, (iv) Most learning is carried out in an unsupervised manner. SSL with a contrastive loss satisfies the third condition as it does not require labelled data and it introduces robustness to observed perturbations of objects, which occur naturally as objects or observer move in 3d and with variable lighting over time. We propose a contrastive hinge based loss whose error involves simple local computations satisfying (ii), as opposed to the standard contrastive losses employed in the literature, which do not lend themselves easily to implementation in a network architecture due to complex computations involving ratios and inner products. Furthermore we show that learning can be performed with one of two more plausible alternatives to backpropagation that satisfy conditions (i) and (ii). The first is difference target propagation (DTP) and the second is layer-wise learning (LL), where each layer is directly connected to a layer computing the loss error. Both methods represent alternatives to the symmetric weight issue of backpropagation. By training convolutional neural networks (CNNs) with SSL and DTP, LL, we find that our proposed framework achieves comparable performance to standard BP learning downstream linear classifier evaluation of the learned embeddings.