# Competitive plasticity to reduce the energetic costs of learning

arXiv: https://arxiv.org/abs/2304.02594 · HTML: https://mlsanity.com/paper/2304.02594/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 44 references that are in the map.
- First posted on arXiv 4 Apr 2023.

## Builds on (most cited first, 9 of 9)

- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://mlsanity.com/paper/1602.01528/) (2016)
- [Dendritic cortical microcircuits approximate the backpropagation algorithm](https://mlsanity.com/paper/1810.11393/) (2018)
- [Learning where to learn: Gradient sparsity in meta and continual learning](https://mlsanity.com/paper/2110.14402/) (2021)
- [Full deep neural network training on a pruned weight budget](https://mlsanity.com/paper/1806.06949/) (2018)
- [Estimating the energy requirements for long term memory formation](https://mlsanity.com/paper/2301.09565/) (2023)
- [Metabolic constraints on synaptic learning and memory](https://mlsanity.com/paper/1910.07414/) (2019)
- [A learning rule balancing energy consumption and information maximization in a feed-forward neuronal network](https://mlsanity.com/paper/2103.06562/) (2021)
- [Lazy learning: a biologically-inspired plasticity rule for fast and energy efficient synaptic plasticity](https://mlsanity.com/paper/2303.16067/) (2023)

## Built on by (most cited first, 1 of 1)

- [Mistake gating leads to energy and memory efficient continual learning](https://mlsanity.com/paper/2604.14336/) (2026)

## Abstract (arXiv)

The brain is not only constrained by energy needed to fuel computation, but it is also constrained by energy needed to form memories. Experiments have shown that learning simple conditioning tasks already carries a significant metabolic cost. Yet, learning a task like MNIST to 95% accuracy appears to require at least 10^{8} synaptic updates. Therefore the brain has likely evolved to be able to learn using as little energy as possible. We explored the energy required for learning in feedforward neural networks. Based on a parsimonious energy model, we propose two plasticity restricting algorithms that save energy: 1) only modify synapses with large updates, and 2) restrict plasticity to subsets of synapses that form a path through the network. Combining these two methods leads to substantial energy savings while only incurring a small increase in learning time. In biology networks are often much larger than the task requires. In particular in that case, large savings can be achieved. Thus competitively restricting plasticity helps to save metabolic energy associated to synaptic plasticity. The results might lead to a better understanding of biological plasticity and a better match between artificial and biological learning. Moreover, the algorithms might also benefit hardware because in electronics memory storage is energetically costly as well.