# Algorithmic insights on continual learning from fruit flies

arXiv: https://arxiv.org/abs/2107.07617 · HTML: https://mlsanity.com/paper/2107.07617/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 80 references that are in the map.
- First posted on arXiv 15 Jul 2021.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [Towards Robust Evaluations of Continual Learning](https://mlsanity.com/paper/1805.09733/) (2018)
- [How Can We Be So Dense? The Benefits of Using Highly Sparse Representations](https://mlsanity.com/paper/1903.11257/) (2019)
- [Bio-Inspired Hashing for Unsupervised Similarity Search](https://mlsanity.com/paper/2001.04907/) (2020)
- [Expressivity of expand-and-sparsify representations](https://mlsanity.com/paper/2006.03741/) (2020)

## Built on by (most cited first, 11 of 11)

- [Discrete Key-Value Bottleneck](https://mlsanity.com/paper/2207.11240/) (2022)
- [Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation](https://mlsanity.com/paper/2205.09029/) (2022)
- [Sparse Distributed Memory is a Continual Learner](https://mlsanity.com/paper/2303.11934/) (2023)
- [Elephant Neural Networks: Born to Be a Continual Learner](https://mlsanity.com/paper/2310.01365/) (2023)
- [Statistical mechanics of continual learning: variational principle and mean-field potential](https://mlsanity.com/paper/2212.02846/) (2022)
- [Lifelong Intelligence Beyond the Edge using Hyperdimensional Computing](https://mlsanity.com/paper/2403.04759/) (2024)
- [Exploiting Large Neuroimaging Datasets to Create Connectome-Constrained Approaches for more Robust, Efficient, and Adaptable Artificial Intelligence](https://mlsanity.com/paper/2305.17300/) (2023)
- [Hard ASH: Sparsity and the right optimizer make a continual learner](https://mlsanity.com/paper/2404.17651/) (2024)
- [Task agnostic continual learning with Pairwise layer architecture](https://mlsanity.com/paper/2405.13632/) (2024)
- [Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functions](https://mlsanity.com/paper/2509.19159/) (2025)
- [Replay in the Silent Degrees of Freedom: Continual Learning Without an Offline Phase](https://mlsanity.com/paper/2609.31630/) (2026)

## Abstract (arXiv)

Continual learning in computational systems is challenging due to catastrophic forgetting. We discovered a two layer neural circuit in the fruit fly olfactory system that addresses this challenge by uniquely combining sparse coding and associative learning. In the first layer, odors are encoded using sparse, high dimensional representations, which reduces memory interference by activating non overlapping populations of neurons for different odors. In the second layer, only the synapses between odor activated neurons and the output neuron associated with the odor are modified during learning; the rest of the weights are frozen to prevent unrelated memories from being overwritten. We show empirically and analytically that this simple and lightweight algorithm significantly boosts continual learning performance. The fly associative learning algorithm is strikingly similar to the classic perceptron learning algorithm, albeit two modifications, which we show are critical for reducing catastrophic forgetting. Overall, fruit flies evolved an efficient lifelong learning algorithm, and circuit mechanisms from neuroscience can be translated to improve machine computation.