# Synaptic Dynamics Realize First-order Adaptive Learning and Weight Symmetry

arXiv: https://arxiv.org/abs/2212.09440 · HTML: https://mlsanity.com/paper/2212.09440/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 1 Dec 2022.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [Dendritic cortical microcircuits approximate the backpropagation algorithm](https://mlsanity.com/paper/1810.11393/) (2018)
- [Deep Learning without Weight Transport](https://mlsanity.com/paper/1904.05391/) (2019)
- [Towards deep learning with segregated dendrites](https://mlsanity.com/paper/1610.00161/) (2016)
- [Biologically plausible deep learning -- but how far can we go with shallow networks?](https://mlsanity.com/paper/1905.04101/) (2019)
- [Backpropagated Neighborhood Aggregation for Accurate Training of Spiking Neural Networks](https://mlsanity.com/paper/2107.06861/) (2021)
- [Learning to Adapt by Minimizing Discrepancy](https://mlsanity.com/paper/1711.11542/) (2017)
- [BioLeaF: A Bio-plausible Learning Framework for Training of Spiking Neural Networks](https://mlsanity.com/paper/2111.13188/) (2021)
- [A Computational Framework of Cortical Microcircuits Approximates Sign-concordant Random Backpropagation](https://mlsanity.com/paper/2205.07292/) (2022)

## Abstract (arXiv)

Gradient-based first-order adaptive optimization methods such as the Adam optimizer are prevalent in training artificial networks, achieving the state-of-the-art results. This work attempts to answer the question whether it is viable for biological neural systems to adopt such optimization methods. To this end, we demonstrate a realization of the Adam optimizer using biologically-plausible mechanisms in synapses. The proposed learning rule has clear biological correspondence, runs continuously in time, and achieves performance to comparable Adam's. In addition, we present a new approach, inspired by the predisposition property of synapses observed in neuroscience, to circumvent the biological implausibility of the weight transport problem in backpropagation (BP). With only local information and no separate training phases, this method establishes and maintains weight symmetry in the forward and backward signaling paths, and is applicable to the proposed biologically plausible Adam learning rule. These mechanisms may shed light on the way in which biological synaptic dynamics facilitate learning.