# Deep learning with asymmetric connections and Hebbian updates

arXiv: https://arxiv.org/abs/1812.07965 · HTML: https://mlsanity.com/paper/1812.07965/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2018.

## Built on by (most cited first, 16 of 16)

- [Predictive Coding: a Theoretical and Experimental Review](https://mlsanity.com/paper/2107.12979/) (2021)
- [Holomorphic Equilibrium Propagation Computes Exact Gradients Through Finite Size Oscillations](https://mlsanity.com/paper/2209.00530/) (2022)
- [Reverse Differentiation via Predictive Coding](https://mlsanity.com/paper/2103.04689/) (2021)
- [Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons](https://mlsanity.com/paper/2110.14549/) (2021)
- [Meta-Learning Biologically Plausible Plasticity Rules with Random Feedback Pathways](https://mlsanity.com/paper/2210.16414/) (2022)
- [Applications of the Free Energy Principle to Machine Learning and Neuroscience](https://mlsanity.com/paper/2107.00140/) (2021)
- [Predictive Coding as a Neuromorphic Alternative to Backpropagation: A Critical Evaluation](https://mlsanity.com/paper/2304.02658/) (2023)
- [Spiking Neural Networks and Bio-Inspired Supervised Deep Learning: A Survey](https://mlsanity.com/paper/2307.16235/) (2023)
- [Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks](https://mlsanity.com/paper/2109.15089/) (2021)
- [Distinguishing Learning Rules with Brain Machine Interfaces](https://mlsanity.com/paper/2206.13448/) (2022)
- [Pretraining with Random Noise for Fast and Robust Learning without Weight Transport](https://mlsanity.com/paper/2405.16731/) (2024)
- [Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules](https://mlsanity.com/paper/2206.00823/) (2022)
- [Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry](https://mlsanity.com/paper/2306.04810/) (2023)
- [Polarity is all you need to learn and transfer faster](https://mlsanity.com/paper/2303.17589/) (2023)
- [Representation Learning in a Decomposed Encoder Design for Bio-inspired Hebbian Learning](https://mlsanity.com/paper/2401.08603/) (2023)
- [Closed-Form Feedback-Free Learning with Forward Projection](https://mlsanity.com/paper/2501.16476/) (2025)

## Abstract (arXiv)

We show that deep networks can be trained using Hebbian updates yielding similar performance to ordinary back-propagation on challenging image datasets. To overcome the unrealistic symmetry in connections between layers, implicit in back-propagation, the feedback weights are separate from the feedforward weights. The feedback weights are also updated with a local rule, the same as the feedforward weights - a weight is updated solely based on the product of activity of the units it connects. With fixed feedback weights as proposed in Lillicrap et. al (2016) performance degrades quickly as the depth of the network increases. If the feedforward and feedback weights are initialized with the same values, as proposed in Zipser and Rumelhart (1990), they remain the same throughout training thus precisely implementing back-propagation. We show that even when the weights are initialized differently and at random, and the algorithm is no longer performing back-propagation, performance is comparable on challenging datasets. We also propose a cost function whose derivative can be represented as a local Hebbian update on the last layer. Convolutional layers are updated with tied weights across space, which is not biologically plausible. We show that similar performance is achieved with untied layers, also known as locally connected layers, corresponding to the connectivity implied by the convolutional layers, but where weights are untied and updated separately. In the linear case we show theoretically that the convergence of the error to zero is accelerated by the update of the feedback weights.