# A theory of capacity and sparse neural encoding

arXiv: https://arxiv.org/abs/2102.10148 · HTML: https://mlsanity.com/paper/2102.10148/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 74 references that are in the map.
- First posted on arXiv 19 Feb 2021.

## Builds on (most cited first, 2 of 2)

- [The capacity of feedforward neural networks](https://mlsanity.com/paper/1901.00434/) (2019)
- [A Theory of Local Learning, the Learning Channel, and the Optimality of Backpropagation](https://mlsanity.com/paper/1506.06472/) (2015)

## Built on by (most cited first, 1 of 1)

- [Tractability from overparametrization: The example of the negative perceptron](https://mlsanity.com/paper/2110.15824/) (2021)

## Abstract (arXiv)

Motivated by biological considerations, we study sparse neural maps from an input layer to a target layer with sparse activity, and specifically the problem of storing K input-target associations (x,y), or memories, when the target vectors y are sparse. We mathematically prove that K undergoes a phase transition and that in general, and somewhat paradoxically, sparsity in the target layers increases the storage capacity of the map. The target vectors can be chosen arbitrarily, including in random fashion, and the memories can be both encoded and decoded by networks trained using local learning rules, including the simple Hebb rule. These results are robust under a variety of statistical assumptions on the data. The proofs rely on elegant properties of random polytopes and sub-gaussian random vector variables. Open problems and connections to capacity theories and polynomial threshold maps are discussed.