# Oja's plasticity rule overcomes several challenges of training neural networks under biological constraints

arXiv: https://arxiv.org/abs/2408.08408 · HTML: https://mlsanity.com/paper/2408.08408/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 55 references that are in the map.
- First posted on arXiv 15 Aug 2024.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [A Simple Way to Initialize Recurrent Networks of Rectified Linear Units](https://mlsanity.com/paper/1504.00941/) (2015)
- [All you need is a good init](https://mlsanity.com/paper/1511.06422/) (2015)
- [Deep Learning without Weight Transport](https://mlsanity.com/paper/1904.05391/) (2019)
- [On orthogonality and learning recurrent networks with long term dependencies](https://mlsanity.com/paper/1702.00071/) (2017)
- [How Important is Weight Symmetry in Backpropagation?](https://mlsanity.com/paper/1510.05067/) (2015)
- [Recurrent Orthogonal Networks and Long-Memory Tasks](https://mlsanity.com/paper/1602.06662/) (2016)
- [Fast and accurate optimization on the orthogonal manifold without retraction](https://mlsanity.com/paper/2102.07432/) (2021)
- [A Hebbian/Anti-Hebbian Neural Network for Linear Subspace Learning: A Derivation from Multidimensional Scaling of Streaming Data](https://mlsanity.com/paper/1503.00669/) (2015)
- [Two Routes to Scalable Credit Assignment without Weight Symmetry](https://mlsanity.com/paper/2003.01513/) (2020)
- [Meta-Learning Biologically Plausible Plasticity Rules with Random Feedback Pathways](https://mlsanity.com/paper/2210.16414/) (2022)
- [Regularizing RNNs by Stabilizing Activations](https://mlsanity.com/paper/1511.08400/) (2015)
- [Cortical microcircuits as gated-recurrent neural networks](https://mlsanity.com/paper/1711.02448/) (2017)

## Built on by (most cited first, 1 of 1)

- [Constrained Hebbian Learning Supports Efficient Representational Allocation under Structural Constraints](https://mlsanity.com/paper/2607.16027/) (2026)

## Abstract (arXiv)

Deep neural networks have achieved impressive performance through carefully engineered training strategies. Nonetheless, such methods lack parallels in biological neural circuits, relying heavily on non-local credit assignment, precise initialization, normalization layers, batch processing, and large datasets. Biologically plausible plasticity rules, such as random feedback alignment, often suffer from instability and unbounded weight growth without these engineered methods, while Hebbian-type schemes fail to provide goal-oriented credit. In this study, we demonstrate that incorporating Oja's plasticity rule into error-driven training yields stable, efficient learning in feedforward and recurrent architectures, obviating the need for carefully engineered tricks. Our results show that Oja's rule preserves richer activation subspaces, mitigates exploding or vanishing signals, and improves short-term memory in recurrent networks. Notably, meta-learned local plasticity rules incorporating Oja's principle not only match but surpass standard backpropagation in data-scarce regimes. These findings reveal a biologically grounded pathway bridging engineered deep networks and plausible synaptic mechanisms.