# A theory for the sparsity emerged in the Forward Forward algorithm

arXiv: https://arxiv.org/abs/2311.05667 · HTML: https://mlsanity.com/paper/2311.05667/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 5 references that are in the map.
- First posted on arXiv 9 Nov 2023.

## Builds on (most cited first, 3 of 3)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Emergent representations in networks trained with the Forward-Forward algorithm](https://mlsanity.com/paper/2305.18353/) (2023)

## Built on by (most cited first, 7 of 7)

- [Self-Contrastive Forward-Forward Algorithm](https://mlsanity.com/paper/2409.11593/) (2024)
- [Training Convolutional Neural Networks with the Forward-Forward algorithm](https://mlsanity.com/paper/2312.14924/) (2023)
- [Resource-efficient Medical Image Analysis with Self-adapting Forward-Forward Networks](https://mlsanity.com/paper/2406.14038/) (2024)
- [A Contrastive Symmetric Forward-Forward Algorithm (SFFA) for Continual Learning Tasks](https://mlsanity.com/paper/2409.07387/) (2024)
- [Emerging NeoHebbian Dynamics in Forward-Forward Learning: Implications for Neuromorphic Computing](https://mlsanity.com/paper/2406.16479/) (2024)
- [On the Improvement of Generalization and Stability of Forward-Only Learning via Neural Polarization](https://mlsanity.com/paper/2408.09210/) (2024)
- [Forward-Only Convolutional Neural Networks with Learnable Channel-Class Assignment](https://mlsanity.com/paper/2606.09928/) (2026)

## Abstract (arXiv)

This report explores the theory that explains the high sparsity phenomenon citeptosato2023emergent observed in the forward-forward algorithm citephinton2022forward. The two theorems proposed predict the sparsity changes of a single data point's activation in two cases: Theorem reftheorem:1: Decrease the goodness of the whole batch. Theorem reftheorem:2: Apply the complete forward forward algorithm to decrease the goodness for negative data and increase the goodness for positive data. The theory aligns well with the experiments tested on the MNIST dataset.