# Leader-Follower Neural Networks with Local Error Signals Inspired by Complex Collectives

arXiv: https://arxiv.org/abs/2310.07885 · HTML: https://mlsanity.com/paper/2310.07885/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 54 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup](https://mlsanity.com/paper/2009.06962/) (2020)
- [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016)
- [Training Neural Networks with Local Error Signals](https://mlsanity.com/paper/1901.06656/) (2019)
- [Greedy Layerwise Learning Can Scale to ImageNet](https://mlsanity.com/paper/1812.11446/) (2018)
- [Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures](https://mlsanity.com/paper/1807.04587/) (2018)
- [Putting An End to End-to-End: Gradient-Isolated Learning of Representations](https://mlsanity.com/paper/1905.11786/) (2019)
- [Towards an integration of deep learning and neuroscience](https://mlsanity.com/paper/1606.03813/) (2016)
- [Gradients without Backpropagation](https://mlsanity.com/paper/2202.08587/) (2022)
- [Scaling Forward Gradient With Local Losses](https://mlsanity.com/paper/2210.03310/) (2022)
- [LoCo: Local Contrastive Representation Learning](https://mlsanity.com/paper/2008.01342/) (2020)
- [Local plasticity rules can learn deep representations using self-supervised contrastive predictions](https://mlsanity.com/paper/2010.08262/) (2020)
- [Deep supervised learning using local errors](https://mlsanity.com/paper/1711.06756/) (2017)
- [Gated Linear Networks](https://mlsanity.com/paper/1910.01526/) (2019)
- [DLME: Deep Local-flatness Manifold Embedding](https://mlsanity.com/paper/2207.03160/) (2022)
- [Multi-manifold Attention for Vision Transformers](https://mlsanity.com/paper/2207.08569/) (2022)
- [Searching for collective behavior in a network of real neurons](https://mlsanity.com/paper/1306.3061/) (2013)
- [UPANets: Learning from the Universal Pixel Attention Networks](https://mlsanity.com/paper/2103.08640/) (2021)
- [Collective behavior of place and non-place neurons in the hippocampal network](https://mlsanity.com/paper/1612.08935/) (2016)

## Built on by (most cited first, 1 of 1)

- [Survey and Evaluation of Converging Architecture in LLMs based on Footsteps of Operations](https://mlsanity.com/paper/2410.11381/) (2024)

## Abstract (arXiv)

The collective behavior of a network with heterogeneous, resource-limited information processing units (e.g., group of fish, flock of birds, or network of neurons) demonstrates high self-organization and complexity. These emergent properties arise from simple interaction rules where certain individuals can exhibit leadership-like behavior and influence the collective activity of the group. Motivated by the intricacy of these collectives, we propose a neural network (NN) architecture inspired by the rules observed in nature's collective ensembles. This NN structure contains workers that encompass one or more information processing units (e.g., neurons, filters, layers, or blocks of layers). Workers are either leaders or followers, and we train a leader-follower neural network (LFNN) by leveraging local error signals and optionally incorporating backpropagation (BP) and global loss. We investigate worker behavior and evaluate LFNNs through extensive experimentation. Our LFNNs trained with local error signals achieve significantly lower error rates than previous BP-free algorithms on MNIST and CIFAR-10 and even surpass BP-enabled baselines. In the case of ImageNet, our LFNN-l demonstrates superior scalability and outperforms previous BP-free algorithms by a significant margin.