# The Forward-Forward Algorithm: Some Preliminary Investigations

arXiv: https://arxiv.org/abs/2212.13345 · HTML: https://mlsanity.com/paper/2212.13345/

## Key facts

- Cited by 266 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 43 references that are in the map.
- First posted on arXiv 27 Dec 2022.

## Builds on (most cited first, 22 of 22)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Using Fast Weights to Attend to the Recent Past](https://mlsanity.com/paper/1610.06258/) (2016)
- [How to represent part-whole hierarchies in a neural network](https://mlsanity.com/paper/2102.12627/) (2021)
- [Putting An End to End-to-End: Gradient-Isolated Learning of Representations](https://mlsanity.com/paper/1905.11786/) (2019)
- [Towards deep learning with segregated dendrites](https://mlsanity.com/paper/1610.00161/) (2016)
- [Scaling Forward Gradient With Local Losses](https://mlsanity.com/paper/2210.03310/) (2022)
- [Training End-to-End Analog Neural Networks with Equilibrium Propagation](https://mlsanity.com/paper/2006.01981/) (2020)
- [Signal Propagation: A Framework for Learning and Inference In a Forward Pass](https://mlsanity.com/paper/2204.01723/) (2022)
- [Error Forward-Propagation: Reusing Feedforward Connections to Propagate Errors in Deep Learning](https://mlsanity.com/paper/1808.03357/) (2018)
- [Teaching with Commentaries](https://mlsanity.com/paper/2011.03037/) (2020)
- [Generative Trees: Adversarial and Copycat](https://mlsanity.com/paper/2201.11205/) (2022)

## Built on by (most cited first, 30 of 266)

- [Fine-Tuning Language Models with Just Forward Passes](https://mlsanity.com/paper/2305.17333/) (2023)
- [Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark](https://mlsanity.com/paper/2402.11592/) (2024)
- [Uncovering mesa-optimization algorithms in Transformers](https://mlsanity.com/paper/2309.05858/) (2023)
- [Better than classical? The subtle art of benchmarking quantum machine learning models](https://mlsanity.com/paper/2403.07059/) (2024)
- [White-Box Transformers via Sparse Rate Reduction](https://mlsanity.com/paper/2306.01129/) (2023)
- [DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training](https://mlsanity.com/paper/2310.02025/) (2023)
- [FLEX: Continuous Agent Evolution via Forward Learning from Experience](https://mlsanity.com/paper/2511.06449/) (2025)
- [Training of Physical Neural Networks](https://mlsanity.com/paper/2406.03372/) (2024)
- [Backpropagation-free Training of Deep Physical Neural Networks](https://mlsanity.com/paper/2304.11042/) (2023)
- [The Predictive Forward-Forward Algorithm](https://mlsanity.com/paper/2301.01452/) (2023)
- [On quantum backpropagation, information reuse, and cheating measurement collapse](https://mlsanity.com/paper/2305.13362/) (2023)
- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)
- [Benchmarking Predictive Coding Networks -- Made Simple](https://mlsanity.com/paper/2407.01163/) (2024)
- [SymBa: Symmetric Backpropagation-Free Contrastive Learning with Forward-Forward Algorithm for Optimizing Convergence](https://mlsanity.com/paper/2303.08418/) (2023)
- [FwdLLM: Efficient FedLLM using Forward Gradient](https://mlsanity.com/paper/2308.13894/) (2023)
- [DPZero: Private Fine-Tuning of Language Models without Backpropagation](https://mlsanity.com/paper/2310.09639/) (2023)
- [Convolutional Channel-wise Competitive Learning for the Forward-Forward Algorithm](https://mlsanity.com/paper/2312.12668/) (2023)
- [AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning](https://mlsanity.com/paper/2406.18060/) (2024)
- [Beyond Geometry: Comparing the Temporal Structure of Computation in Neural Circuits with Dynamical Similarity Analysis](https://mlsanity.com/paper/2306.10168/) (2023)
- [Blockwise Self-Supervised Learning at Scale](https://mlsanity.com/paper/2302.01647/) (2023)
- [ChatGPT: More than a Weapon of Mass Deception, Ethical challenges and responses from the Human-Centered Artificial Intelligence (HCAI) perspective](https://mlsanity.com/paper/2304.11215/) (2023)
- [Layer Collaboration in the Forward-Forward Algorithm](https://mlsanity.com/paper/2305.12393/) (2023)
- [Forward-Forward Training of an Optical Neural Network](https://mlsanity.com/paper/2305.19170/) (2023)
- [ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models](https://mlsanity.com/paper/2310.02998/) (2023)
- [The Cascaded Forward Algorithm for Neural Network Training](https://mlsanity.com/paper/2303.09728/) (2023)
- [Learning to (Learn at Test Time)](https://mlsanity.com/paper/2310.13807/) (2023)
- [Self-Contrastive Forward-Forward Algorithm](https://mlsanity.com/paper/2409.11593/) (2024)
- [Zero-th Order Algorithm for Softmax Attention Optimization](https://mlsanity.com/paper/2307.08352/) (2023)
- [Efficient Model Editing with Task-Localized Sparse Fine-tuning](https://mlsanity.com/paper/2504.02620/) (2025)
- [Thermodynamic AI and the fluctuation frontier](https://mlsanity.com/paper/2302.06584/) (2023)

## Abstract (arXiv)

The aim of this paper is to introduce a new learning procedure for neural networks and to demonstrate that it works well enough on a few small problems to be worth further investigation. The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself. Each layer has its own objective function which is simply to have high goodness for positive data and low goodness for negative data. The sum of the squared activities in a layer can be used as the goodness but there are many other possibilities, including minus the sum of the squared activities. If the positive and negative passes could be separated in time, the negative passes could be done offline, which would make the learning much simpler in the positive pass and allow video to be pipelined through the network without ever storing activities or stopping to propagate derivatives.