# Hebbian learning with gradients: Hebbian convolutional neural networks with modern deep learning frameworks

arXiv: https://arxiv.org/abs/2107.01729 · HTML: https://mlsanity.com/paper/2107.01729/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jul 2021.

## Built on by (most cited first, 19 of 19)

- [Convolutional Channel-wise Competitive Learning for the Forward-Forward Algorithm](https://mlsanity.com/paper/2312.12668/) (2023)
- [Self-Contrastive Forward-Forward Algorithm](https://mlsanity.com/paper/2409.11593/) (2024)
- [Is Bio-Inspired Learning Better than Backprop? Benchmarking Bio Learning vs. Backprop](https://mlsanity.com/paper/2212.04614/) (2022)
- [Spiking Neural Networks and Bio-Inspired Supervised Deep Learning: A Survey](https://mlsanity.com/paper/2307.16235/) (2023)
- [Distance-Forward Learning: Enhancing the Forward-Forward Algorithm Towards High-Performance On-Chip Learning](https://mlsanity.com/paper/2408.14925/) (2024)
- [Synaptic Plasticity Models and Bio-Inspired Unsupervised Deep Learning: A Survey](https://mlsanity.com/paper/2307.16236/) (2023)
- [Advancing the Biological Plausibility and Efficacy of Hebbian Convolutional Neural Networks](https://mlsanity.com/paper/2501.17266/) (2025)
- [Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation](https://mlsanity.com/paper/2308.12053/) (2023)
- [OscNet: Machine Learning on CMOS Oscillator Networks](https://mlsanity.com/paper/2502.07192/) (2025)
- [HCL-FF: Hierarchical and Contrastive Learning for Forward-Forward Algorithm](https://mlsanity.com/paper/2605.24797/) (2026)
- [Unsupervised End-to-End Training with a Self-Defined Target](https://mlsanity.com/paper/2403.12116/) (2024)
- [Towards Unsupervised Representation Learning: Learning, Evaluating and Transferring Visual Representations](https://mlsanity.com/paper/2312.00101/) (2023)
- [The Cooperative Network Architecture: Learning Structured Networks as Representation of Sensory Patterns](https://mlsanity.com/paper/2407.05650/) (2024)
- [Memory Networks: Towards Fully Biologically Plausible Learning](https://mlsanity.com/paper/2409.17282/) (2024)
- [Meta-Representational Predictive Coding: Neuroscience-Informed Self-Supervised Learning](https://mlsanity.com/paper/2503.21796/) (2025)
- [Rethinking Hebbian Principle: Low-Dimensional Structural Projection for Unsupervised Learning](https://mlsanity.com/paper/2510.14810/) (2025)
- [Relational Intervention During Functional Collapse in Large Language Models: A Lexical-Statistical Ablation and a Structure x Register Factorial](https://mlsanity.com/paper/2606.00935/) (2026)
- [Orthogonal Dendritic Intrinsic Networks: An Architecture for Significance-Ordered, Orthogonal Latent Spaces](https://mlsanity.com/paper/2607.05653/) (2026)
- [Constrained Hebbian Learning Supports Efficient Representational Allocation under Structural Constraints](https://mlsanity.com/paper/2607.16027/) (2026)

## Abstract (arXiv)

Deep learning networks generally use non-biological learning methods. By contrast, networks based on more biologically plausible learning, such as Hebbian learning, show comparatively poor performance and difficulties of implementation. Here we show that Hebbian learning in hierarchical, convolutional neural networks can be implemented almost trivially with modern deep learning frameworks, by using specific losses whose gradients produce exactly the desired Hebbian updates. We provide expressions whose gradients exactly implement a plain Hebbian rule (dw ~= xy), Grossberg's instar rule (dw ~= y(x-w)), and Oja's rule (dw ~= y(x-yw)). As an application, we build Hebbian convolutional multi-layer networks for object recognition. We observe that higher layers of such networks tend to learn large, simple features (Gabor-like filters and blobs), explaining the previously reported decrease in decoding performance over successive layers. To combat this tendency, we introduce interventions (denser activations with sparse plasticity, pruning of connections between layers) which result in sparser learned features, massively increase performance, and allow information to increase over successive layers. We hypothesize that more advanced techniques (dynamic stimuli, trace learning, feedback connections, etc.), together with the massive computational boost offered by modern deep learning frameworks, could greatly improve the performance and biological relevance of multi-layer Hebbian networks.