mlsanityData to 7 Oct 2026

The Forward-Forward Algorithm: Some Preliminary Investigations

arXiv 2212.13345 ↗First posted 27 Dec 2022cs.LGcited by 262

Not on the map. Why?

Abstract (arXiv)

The aim of this paper is to introduce a new learning procedure for neural networks and to demonstrate that it works well enough on a few small problems to be worth further investigation. The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself. Each layer has its own objective function which is simply to have high goodness for positive data and low goodness for negative data. The sum of the squared activities in a layer can be used as the goodness but there are many other possibilities, including minus the sum of the squared activities. If the positive and negative passes could be separated in time, the negative passes could be done offline, which would make the learning much simpler in the positive pass and allow video to be pipelined through the network without ever storing activities or stopping to propagate derivatives.

Key facts. Cited by 262 papers in the map (data through 7 Oct 2026). Builds on 22 of its 43 references that are in the map. First posted on arXiv 27 Dec 2022.

Builds on

22 of 43 references.

References · 22

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2016Layer Normalizationcited by 5,579
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  8. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  9. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  10. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  11. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  12. 2016Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagationcited by 251
10 more
  1. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  2. 2021How to represent part-whole hierarchies in a neural networkcited by 123
  3. 2019Putting An End to End-to-End: Gradient-Isolated Learning of Representationscited by 85
  4. 2016Towards deep learning with segregated dendritescited by 71
  5. 2022Scaling Forward Gradient With Local Lossescited by 65
  6. 2020Training End-to-End Analog Neural Networks with Equilibrium Propagationcited by 57
  7. 2022Signal Propagation: A Framework for Learning and Inference In a Forward Passcited by 24
  8. 2018Error Forward-Propagation: Reusing Feedforward Connections to Propagate Errors in Deep Learningcited by 16
  9. 2020Teaching with Commentariescited by 16
  10. 2022Generative Trees: Adversarial and Copycatcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 75; 2024: 63; 2025: 70; 2026: 54.

262 papers.

Sort:

By area:

  1. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  2. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  3. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  4. 2024Better than classical? The subtle art of benchmarking quantum machine learning modelscited by 82
  5. 2023White-Box Transformers via Sparse Rate Reductioncited by 77
  6. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  7. 2025FLEX: Continuous Agent Evolution via Forward Learning from Experiencecited by 51
  8. 2024Training of Physical Neural Networkscited by 47
  9. 2023Backpropagation-free Training of Deep Physical Neural Networkscited by 42
  10. 2023The Predictive Forward-Forward Algorithmcited by 35
  11. 2023On quantum backpropagation, information reuse, and cheating measurement collapsecited by 32
  12. 2023Brain-inspired Computational Intelligence via Predictive Codingcited by 30
38 more
  1. 2024Benchmarking Predictive Coding Networks -- Made Simplecited by 29
  2. 2023SymBa: Symmetric Backpropagation-Free Contrastive Learning with Forward-Forward Algorithm for Optimizing Convergencecited by 27
  3. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  4. 2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 25
  5. 2024AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuningcited by 25
  6. 2023Convolutional Channel-wise Competitive Learning for the Forward-Forward Algorithmcited by 23
  7. 2023Blockwise Self-Supervised Learning at Scalecited by 22
  8. 2023ChatGPT: More than a Weapon of Mass Deception, Ethical challenges and responses from the Human-Centered Artificial Intelligence (HCAI) perspectivecited by 22
  9. 2023Beyond Geometry: Comparing the Temporal Structure of Computation in Neural Circuits with Dynamical Similarity Analysiscited by 21
  10. 2023Layer Collaboration in the Forward-Forward Algorithmcited by 19
  11. 2023Forward-Forward Training of an Optical Neural Networkcited by 19
  12. 2023ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Modelscited by 19
  13. 2023The Cascaded Forward Algorithm for Neural Network Trainingcited by 17
  14. 2023Zero-th Order Algorithm for Softmax Attention Optimizationcited by 16
  15. 2025Efficient Model Editing with Task-Localized Sparse Fine-tuningcited by 16
  16. 2023Thermodynamic AI and the fluctuation frontiercited by 15
  17. 2023How connectivity structure shapes rich and lazy learning in neural circuitscited by 15
  18. 2024Self-Contrastive Forward-Forward Algorithmcited by 15
  19. 2023Forward Learning with Top-Down Feedback: Empirical and Analytical Characterizationcited by 13
  20. 2023Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignmentcited by 13
  21. 2023Training Convolutional Neural Networks with the Forward-Forward algorithmcited by 13
  22. 2023Tensor-Compressed Back-Propagation-Free Training for (Physics-Informed) Neural Networkscited by 12
  23. 2023Learning to (Learn at Test Time)cited by 12
  24. 2023Representation Learning via Manifold Flattening and Reconstructioncited by 11
  25. 2023One Forward is Enough for Neural Network Training via Likelihood Ratio Methodcited by 11
  26. 2023Emergent representations in networks trained with the Forward-Forward algorithmcited by 11
  27. 2023Recasting Continual Learning as Sequence Modelingcited by 11
  28. 2024Training Machine Learning models at the Edge: A Surveycited by 11
  29. 2023Contrastive-Signal-Dependent Plasticity: Self-Supervised Learning in Spiking Neural Circuitscited by 10
  30. 2023The Trifecta: Three simple techniques for training deeper Forward-Forward networkscited by 10
  31. 2024Adapt-∞: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selectioncited by 9
  32. 2025ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Thinkcited by 9
  33. 2025DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretationcited by 9
  34. 2023Computational Dualism and Objective Superintelligencecited by 8
  35. 2023Graph Neural Networks Go Forward-Forwardcited by 8
  36. 2023A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functionscited by 8
  37. 2023Spiking Neural Networks and Bio-Inspired Supervised Deep Learning: A Surveycited by 8
  38. 2023Improving equilibrium propagation without weight symmetry through Jacobian homeostasiscited by 8

50 most cited of 262 All 262 (JSON)