# Ubiquity of Emergent Hebbian Dynamics in Regularized Learning

arXiv: https://arxiv.org/abs/2505.18069 · HTML: https://mlsanity.com/paper/2505.18069/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 46 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 13 of 13)

- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [Deep Learning without Weight Transport](https://mlsanity.com/paper/1904.05391/) (2019)
- [Biologically-plausible learning algorithms can scale to large datasets](https://mlsanity.com/paper/1811.03567/) (2018)
- [Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent](https://mlsanity.com/paper/2012.03636/) (2020)
- [Towards Scaling Difference Target Propagation by Learning Backprop Targets](https://mlsanity.com/paper/2201.13415/) (2022)
- [Equivalence of Equilibrium Propagation and Recurrent Backpropagation](https://mlsanity.com/paper/1711.08416/) (2017)
- [Generalization of Equilibrium Propagation to Vector Field Dynamics](https://mlsanity.com/paper/1808.04873/) (2018)
- [Why do similarity matching objectives lead to Hebbian/anti-Hebbian networks?](https://mlsanity.com/paper/1703.07914/) (2017)
- [Formation of Representations in Neural Networks](https://mlsanity.com/paper/2410.03006/) (2024)
- [Nonlinear Hebbian learning as a unifying principle in receptive field formation](https://mlsanity.com/paper/1601.00701/) (2016)
- [How Biological Synapses Self-Assemble Gradient Learning](https://mlsanity.com/paper/2412.20018/) (2024)
- [Heterosynaptic Circuits Are Universal Gradient Machines](https://mlsanity.com/paper/2505.02248/) (2025)

## Abstract (arXiv)

Hebbian and anti-Hebbian plasticity are widely observed in the brain and are classically modeled as mechanistic, local homosynaptic rules stabilized by homeostatic constraints. This raises an identifiability question: does observing Hebbian/anti-Hebbian structure in synaptic updates uniquely imply an underlying Hebbian computation? We identify an alternative, emergent route. We show that near stationarity, L2 weight decay generically drives the learning-signal component of many update rules to align with a Hebbian direction, with alignment increasing monotonically with decay strength. This Hebbian-like signature is not specific to SGD and can arise even for non-learning or random update rules long before learning has ceased. We further show that stochastic noise in the learning signal can induce anti-Hebbian alignment, yielding a simple tradeoff with weight decay and a phase boundary in regression settings. These mechanisms do not replace standard Hebbian theory; they can coexist with genuine Hebbian plasticity and complicate the interpretation of synaptic measurements, motivating experiments that distinguish mechanistic Hebbian computation from emergent Hebbian signatures.