mlsanityData to 14 Oct 2026

Hebbian and Gradient-based Plasticity Enables Robust Memory and Rapid Learning in RNNs

arXiv 2302.03235 ↗First posted 7 Feb 2023cs.NE cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Rapidly learning from ongoing experiences and remembering past events with a flexible memory system are two core capacities of biological intelligence. While the underlying neural mechanisms are not fully understood, various evidence supports that synaptic plasticity plays a critical role in memory formation and fast learning. Inspired by these results, we equip Recurrent Neural Networks (RNNs) with plasticity rules to enable them to adapt their parameters according to ongoing experiences. In addition to the traditional local Hebbian plasticity, we propose a global, gradient-based plasticity rule, which allows the model to evolve towards its self-determined target. Our models show promising results on sequential and associative memory tasks, illustrating their ability to robustly form and retain memories. In the meantime, these models can cope with many challenging few-shot learning problems. Comparing different plasticity rules under the same framework shows that Hebbian plasticity is well-suited for several memory and associative learning tasks; however, it is outperformed by gradient-based plasticity on few-shot regression tasks which require the model to infer the underlying mapping. Code is available at https://github.com/yuvenduan/PlasticRNNs.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 30 of its 63 references that are in the map. First posted on arXiv 7 Feb 2023.

Builds on

30 of 63 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  4. 2016Matching Networks for One Shot Learningcited by 2,255
  5. 2016HyperNetworkscited by 1,027
  6. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  7. 2016Learning to learn by gradient descent by gradient descentcited by 620
  8. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  9. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  10. 2018Meta-learning with differentiable closed-form solverscited by 397
  11. 2018DropBlock: A regularization method for convolutional networkscited by 262
  12. 2016Using Fast Weights to Attend to the Recent Pastcited by 170
18 more
  1. 2020A Survey of Deep Meta-Learningcited by 100
  2. 2018Dendritic cortical microcircuits approximate the backpropagation algorithmcited by 97
  3. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 89
  4. 2018Evolved Policy Gradientscited by 88
  5. 2018Differentiable plasticity: training plastic neural networks with backpropagationcited by 84
  6. 2016Towards an integration of deep learning and neurosciencecited by 78
  7. 2020Discovering Reinforcement Learning Algorithmscited by 77
  8. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  9. 2019Meta-Learning via Learned Losscited by 60
  10. 2020Learning Associative Inference Using Fast Weight Memorycited by 51
  11. 2021Rectifying the Shortcut Learning of Background for Few-Shot Learningcited by 51
  12. 2021Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learningcited by 49
  13. 2020Meta-Learning through Hebbian Plasticity in Random Networkscited by 47
  14. 2019Metalearned Neural Memorycited by 44
  15. 2020Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticitycited by 41
  16. 2021Biological learning in key-value memory networkscited by 18
  17. 2019Meta-Learning Deep Energy-Based Memory Modelscited by 11
  18. 2022Short-Term Plasticity Neurons Learning to Learn and Forgetcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 1; 2026: 2.

4 papers.

  1. 2023Brain-inspired learning in artificial neural networks: a reviewcited by 26
  2. 2025Enabling Robust In-Context Memory and Rapid Task Adaptation in Transformers with Hebbian and Gradient-Based Plasticitycited by 1
  3. 2026RMAAT: Astrocyte-Inspired Memory Compression and Replay for Efficient Long-Context Transformerscited by 1
  4. 2026Continuous Memory Machinescited by 0