# Emergent representations in networks trained with the Forward-Forward algorithm

arXiv: https://arxiv.org/abs/2305.18353 · HTML: https://mlsanity.com/paper/2305.18353/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 69 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models](https://mlsanity.com/paper/2303.10130/) (2023)
- [Accelerating Convolutional Neural Networks via Activation Map Compression](https://mlsanity.com/paper/1812.04056/) (2018)
- [How far can we go without convolution: Improving fully-connected networks](https://mlsanity.com/paper/1511.02580/) (2015)
- [Analyzing biological and artificial neural networks: challenges with opportunities for synergy?](https://mlsanity.com/paper/1810.13373/) (2018)
- [Towards Biologically Plausible Convolutional Networks](https://mlsanity.com/paper/2106.13031/) (2021)

## Built on by (most cited first, 11 of 11)

- [Self-Contrastive Forward-Forward Algorithm](https://mlsanity.com/paper/2409.11593/) (2024)
- [Training Convolutional Neural Networks with the Forward-Forward algorithm](https://mlsanity.com/paper/2312.14924/) (2023)
- [A theory for the sparsity emerged in the Forward Forward algorithm](https://mlsanity.com/paper/2311.05667/) (2023)
- [Backpropagation-free Spiking Neural Networks with the Forward-Forward Algorithm](https://mlsanity.com/paper/2502.20411/) (2025)
- [Resource-efficient Medical Image Analysis with Self-adapting Forward-Forward Networks](https://mlsanity.com/paper/2406.14038/) (2024)
- [A Contrastive Symmetric Forward-Forward Algorithm (SFFA) for Continual Learning Tasks](https://mlsanity.com/paper/2409.07387/) (2024)
- [On the Improvement of Generalization and Stability of Forward-Only Learning via Neural Polarization](https://mlsanity.com/paper/2408.09210/) (2024)
- [Forward-Cooperation-Backward (FCB) learning in a Multi-Encoding Uni-Decoding neural network architecture](https://mlsanity.com/paper/2502.20113/) (2025)
- [Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functions](https://mlsanity.com/paper/2510.06649/) (2025)
- [Emergent Hierarchical Monosemantic Neurons from the Group-Contrastive Forward-Forward Algorithm](https://mlsanity.com/paper/2607.16295/) (2026)
- [The Price of Locality: Why Forward-Forward Underperforms Backpropagation?](https://mlsanity.com/paper/2609.33240/) (2026)

## Abstract (arXiv)

The Backpropagation algorithm has often been criticised for its lack of biological realism. In an attempt to find a more biologically plausible alternative, the recently introduced Forward-Forward algorithm replaces the forward and backward passes of Backpropagation with two forward passes. In this work, we show that the internal representations obtained by the Forward-Forward algorithm can organise into category-specific ensembles exhibiting high sparsity -- composed of a low number of active units. This situation is reminiscent of what has been observed in cortical sensory areas, where neuronal ensembles are suggested to serve as the functional building blocks for perception and action. Interestingly, while this sparse pattern does not typically arise in models trained with standard Backpropagation, it can emerge in networks trained with Backpropagation on the same objective proposed for the Forward-Forward algorithm.