# A Theory of Local Learning, the Learning Channel, and the Optimality of Backpropagation

arXiv: https://arxiv.org/abs/1506.06472 · HTML: https://mlsanity.com/paper/1506.06472/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jun 2015.

## Built on by (most cited first, 11 of 11)

- [Scaling Forward Gradient With Local Losses](https://mlsanity.com/paper/2210.03310/) (2022)
- [Reverse Differentiation via Predictive Coding](https://mlsanity.com/paper/2103.04689/) (2021)
- [Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks](https://mlsanity.com/paper/2103.03725/) (2021)
- [Applications of the Free Energy Principle to Machine Learning and Neuroscience](https://mlsanity.com/paper/2107.00140/) (2021)
- [Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach](https://mlsanity.com/paper/2411.05873/) (2024)
- [Stepping Forward on the Last Mile](https://mlsanity.com/paper/2411.04036/) (2024)
- [A theory of capacity and sparse neural encoding](https://mlsanity.com/paper/2102.10148/) (2021)
- [Tourbillon: a Physically Plausible Neural Architecture](https://mlsanity.com/paper/2107.06424/) (2021)
- [Detecting Pulmonary Coccidioidomycosis (Valley fever) with Deep Convolutional Neural Networks](https://mlsanity.com/paper/2102.00280/) (2021)
- [Feature Alignment as a Generative Process](https://mlsanity.com/paper/2106.12562/) (2021)
- [A Theory of Synaptic Neural Balance: From Local to Global Order](https://mlsanity.com/paper/2405.09688/) (2024)

## Abstract (arXiv)

In a physical neural system, where storage and processing are intimately intertwined, the rules for adjusting the synaptic weights can only depend on variables that are available locally, such as the activity of the pre- and post-synaptic neurons, resulting in local learning rules. A systematic framework for studying the space of local learning rules is obtained by first specifying the nature of the local variables, and then the functional form that ties them together into each learning rule. Such a framework enables also the systematic discovery of new learning rules and exploration of relationships between learning rules and group symmetries. We study polynomial local learning rules stratified by their degree and analyze their behavior and capabilities in both linear and non-linear units and networks. Stacking local learning rules in deep feedforward networks leads to deep local learning. While deep local learning can learn interesting representations, it cannot learn complex input-output functions, even when targets are available for the top layer. Learning complex input-output functions requires local deep learning where target information is communicated to the deep layers through a backward learning channel. The nature of the communicated information about the targets and the structure of the learning channel partition the space of learning algorithms. We estimate the learning channel capacity associated with several algorithms and show that backpropagation outperforms them by simultaneously maximizing the information rate and minimizing the computational cost, even in recurrent networks. The theory clarifies the concept of Hebbian learning, establishes the power and limitations of local learning rules, introduces the learning channel which enables a formal analysis of the optimality of backpropagation, and explains the sparsity of the space of learning rules discovered so far.