mlsanityData to 7 Oct 2026

Context selectivity with dynamic availability enables lifelong continual learning

arXiv 2306.01690 ↗First posted 2 Jun 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

"You never forget how to ride a bike", -- but how is that possible? The brain is able to learn complex skills, stop the practice for years, learn other skills in between, and still retrieve the original knowledge when necessary. The mechanisms of this capability, referred to as lifelong learning (or continual learning, CL), are unknown. We suggest a bio-plausible meta-plasticity rule building on classical work in CL which we summarize in two principles: (i) neurons are context selective, and (ii) a local availability variable partially freezes the plasticity if the neuron was relevant for previous tasks. In a new neuro-centric formalization of these principles, we suggest that neuron selectivity and neuron-wide consolidation is a simple and viable meta-plasticity hypothesis to enable CL in the brain. In simulation, this simple model balances forgetting and consolidation leading to better transfer learning than contemporary CL algorithms on image recognition and natural language processing CL benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 74 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

25 of 74 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2016Wide Residual Networkscited by 3,057
  6. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  7. 2016Progressive Neural Networkscited by 1,332
  8. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  9. 2017Continual Learning with Deep Generative Replaycited by 1,152
  10. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  11. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  12. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
13 more
  1. 2018Reinforced Continual Learningcited by 186
  2. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  3. 2018Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilizationcited by 104
  4. 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
  5. 2021Model Zoo: A Growing "Brain" That Learns Continuallycited by 53
  6. 2020Local plasticity rules can learn deep representations using self-supervised contrastive predictionscited by 47
  7. 2021Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mappingcited by 47
  8. 2021Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Taskscited by 46
  9. 2018Continual Reinforcement Learning with Complex Synapsescited by 30
  10. 2021Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environmentscited by 24
  11. 2022Constrained Predictive Coding as a Biologically Plausible Model of the Cortical Hierarchycited by 17
  12. 2021Continual Learning with Knowledge Transfer for Sentiment Classificationcited by 12
  13. 2022Modelling continual learning in humans with Hebbian context gating and exponentially decaying task signalscited by 12

Built on by

Not cited yet.