mlsanityData to 7 Oct 2026

Sparse Distributed Memory is a Continual Learner

arXiv 2303.11934 ↗First posted 20 Mar 2023cs.NE cond-mat.dis-nn cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Continual learning is a problem for artificial neural networks that their biological counterparts are adept at solving. Building on work using Sparse Distributed Memory (SDM) to connect a core neural circuit with the powerful Transformer model, we create a modified Multi-Layered Perceptron (MLP) that is a strong continual learner. We find that every component of our MLP variant translated from biology is necessary for continual learning. Our solution is also free from any memory replay or task information, and introduces novel methods to train sparse networks that may be broadly applicable.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 43 of its 98 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

43 of 98 references.

References · 43

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  12. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
31 more
  1. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  2. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  3. 2020Hopfield Networks is All You Needcited by 461
  4. 2021Hash Layers For Large Sparse Modelscited by 236
  5. 2016Dense Associative Memory for Pattern Recognitioncited by 230
  6. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  7. 2018Task-Free Continual Learningcited by 219
  8. 2022Patches Are All You Need?cited by 201
  9. 2018Reinforced Continual Learningcited by 186
  10. 2018Towards Robust Evaluations of Continual Learningcited by 148
  11. 2020Sparse GPU Kernels for Deep Learningcited by 138
  12. 2020Large Associative Memory Problem in Neurobiology and Machine Learningcited by 110
  13. 2019Augmenting Self-attention with Persistent Memorycited by 92
  14. 2022A Closer Look at Rehearsal-Free Continual Learningcited by 54
  15. 2022Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Modelscited by 52
  16. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  17. 2017Dense Associative Memory is Robust to Adversarial Inputscited by 45
  18. 2019How Can We Be So Dense? The Benefits of Using Highly Sparse Representationscited by 45
  19. 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 45
  20. 2018Unsupervised Learning by Competing Hidden Unitscited by 44
  21. 2022NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networkscited by 38
  22. 2020SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inferencecited by 30
  23. 2021Attention Approximates Sparse Distributed Memorycited by 29
  24. 2021Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environmentscited by 24
  25. 2018Comparison of non-linear activation functions for deep neural networks on MNIST classification taskcited by 19
  26. 2022Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activationscited by 18
  27. 2021Can a Fruit Fly Learn Word Embeddings?cited by 17
  28. 2019Local Unsupervised Learning for Image Analysiscited by 16
  29. 2020Decentralized Reinforcement Learning: Global Decision-Making via Local Economic Transactionscited by 14
  30. 2020Bio-Inspired Hashing for Unsupervised Similarity Searchcited by 11
  31. 2021Algorithmic insights on continual learning from fruit fliescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 7; 2026: 1.

11 papers.

  1. 2025Key-value memory in the braincited by 22
  2. 2023Elephant Neural Networks: Born to Be a Continual Learnercited by 10
  3. 2025When Truthful Representations Flip Under Deceptive Instructions?cited by 9
  4. 2025Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learningcited by 6
  5. 2024Lifelong Intelligence Beyond the Edge using Hyperdimensional Computingcited by 5
  6. 2025Engram Memory Encoding and Retrieval: A Neurocomputational Perspectivecited by 2
  7. 2024On Design Choices in Similarity-Preserving Sparse Randomized Embeddingscited by 1
  8. 2025Stochastic Engrams for Efficient Continual Learningcited by 1
  9. 2025Comply: Learning Sentences with Complex Weights inspired by Fruit Fly Olfactioncited by 0
  10. 2025Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functionscited by 0
  11. 2026Replay in the Silent Degrees of Freedom: Continual Learning Without an Offline Phasecited by 0