# Self-Contrastive Forward-Forward Algorithm

arXiv: https://arxiv.org/abs/2409.11593 · HTML: https://mlsanity.com/paper/2409.11593/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 66 of its 108 references that are in the map.
- First posted on arXiv 17 Sep 2024.

## Builds on (most cited first, 30 of 66)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Fine-Tuning Language Models with Just Forward Passes](https://mlsanity.com/paper/2305.17333/) (2023)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [Using Fast Weights to Attend to the Recent Past](https://mlsanity.com/paper/1610.06258/) (2016)
- [Normalization Techniques in Training DNNs: Methodology, Analysis and Application](https://mlsanity.com/paper/2009.12836/) (2020)
- [Differentiable plasticity: training plastic neural networks with backpropagation](https://mlsanity.com/paper/1804.02464/) (2018)
- [Robust Inference via Generative Classifiers for Handling Noisy Labels](https://mlsanity.com/paper/1901.11300/) (2019)
- [Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias](https://mlsanity.com/paper/2006.03824/) (2020)
- [Training End-to-End Analog Neural Networks with Equilibrium Propagation](https://mlsanity.com/paper/2006.01981/) (2020)
- [Hebbian Deep Learning Without Feedback](https://mlsanity.com/paper/2209.11883/) (2022)
- [Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures](https://mlsanity.com/paper/2006.12878/) (2020)
- [Training of Physical Neural Networks](https://mlsanity.com/paper/2406.03372/) (2024)
- [Local plasticity rules can learn deep representations using self-supervised contrastive predictions](https://mlsanity.com/paper/2010.08262/) (2020)
- [Unsupervised Learning by Competing Hidden Units](https://mlsanity.com/paper/1806.10181/) (2018)
- [Backpropagation-free Training of Deep Physical Neural Networks](https://mlsanity.com/paper/2304.11042/) (2023)
- [Comparing SNNs and RNNs on Neuromorphic Vision Datasets: Similarities and Differences](https://mlsanity.com/paper/2005.02183/) (2020)
- [Neuromorphic Deep Learning Machines](https://mlsanity.com/paper/1612.05596/) (2016)
- [Self-Supervised Learning with an Information Maximization Criterion](https://mlsanity.com/paper/2209.07999/) (2022)
- [Energy-based learning algorithms for analog computing: a comparative study](https://mlsanity.com/paper/2312.15103/) (2023)
- [The Predictive Forward-Forward Algorithm](https://mlsanity.com/paper/2301.01452/) (2023)
- [SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks](https://mlsanity.com/paper/2107.05747/) (2021)
- [Training an Ising Machine with Equilibrium Propagation](https://mlsanity.com/paper/2305.18321/) (2023)
- [SymBa: Symmetric Backpropagation-Free Contrastive Learning with Forward-Forward Algorithm for Optimizing Convergence](https://mlsanity.com/paper/2303.08418/) (2023)
- [Convolutional Channel-wise Competitive Learning for the Forward-Forward Algorithm](https://mlsanity.com/paper/2312.12668/) (2023)

## Built on by (most cited first, 17 of 17)

- [Mono-Forward: Revisiting Forward-Forward through Objective-Locality Decomposition](https://mlsanity.com/paper/2501.09238/) (2025)
- [Scalable Forward-Forward Algorithm](https://mlsanity.com/paper/2501.03176/) (2025)
- [Covariance-Aware Goodness for Scalable Forward-Forward Learning](https://mlsanity.com/paper/2605.04346/) (2026)
- [HCL-FF: Hierarchical and Contrastive Learning for Forward-Forward Algorithm](https://mlsanity.com/paper/2605.24797/) (2026)
- [On Advancements of the Forward-Forward Algorithm](https://mlsanity.com/paper/2504.21662/) (2025)
- [Learning without Global Backpropagation via Synergistic Information Distillation](https://mlsanity.com/paper/2510.03273/) (2025)
- [Learning Nonlinear Heterogeneity in Physical Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2601.15340/) (2026)
- [Local learning for stable backpropagation-free neural network training towards physical learning](https://mlsanity.com/paper/2603.24790/) (2026)
- [Can Local Learning Match Self-Supervised Backpropagation?](https://mlsanity.com/paper/2601.21683/) (2026)
- [SKYLIGHT: A Scalable Hundred-Channel 3D Photonic In-Memory Tensor Core Architecture for Real-time AI Inference](https://mlsanity.com/paper/2602.19031/) (2026)
- [When Does Margin Clamping Affect Training Variance? Dataset-Dependent Effects in Contrastive Forward-Forward Learning](https://mlsanity.com/paper/2603.00951/) (2026)
- [FFR: Forward-Forward Learning for Regression](https://mlsanity.com/paper/2606.03927/) (2026)
- [Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training](https://mlsanity.com/paper/2606.06539/) (2026)
- [Emergent Hierarchical Monosemantic Neurons from the Group-Contrastive Forward-Forward Algorithm](https://mlsanity.com/paper/2607.16295/) (2026)
- [Learning in Deep Networks under Dale's Constraint](https://mlsanity.com/paper/2608.06963/) (2026)
- [The Price of Locality: Why Forward-Forward Underperforms Backpropagation?](https://mlsanity.com/paper/2609.33240/) (2026)
- [Rescuing Points Trapped Inside Incorrect Decision Regions: A target for Evaluating and Improving Forward-Forward Learning](https://mlsanity.com/paper/2610.16347/) (2026)

## Abstract (arXiv)

Agents that operate autonomously benefit from lifelong learning capabilities. However, compatible training algorithms must comply with the decentralized nature of these systems, which imposes constraints on both the parameter counts and the computational resources. The Forward-Forward (FF) algorithm is one of these. FF relies only on feedforward operations, the same used for inference, for optimizing layer-wise objectives. This purely forward approach eliminates the need for transpose operations required in traditional backpropagation. Despite its potential, FF has failed to reach state-of-the-art performance on most standard benchmark tasks, in part due to unreliable negative data generation methods for unsupervised learning. In this work, we propose the Self-Contrastive Forward-Forward (SCFF) algorithm, a competitive training method aimed at closing this performance gap. Inspired by standard self-supervised contrastive learning for vision tasks, SCFF generates positive and negative inputs applicable across various datasets. The method demonstrates superior performance compared to existing unsupervised local learning algorithms on several benchmark datasets, including MNIST, CIFAR-10, STL-10, and Tiny ImageNet. We extend FF's application to training recurrent neural networks, expanding its utility to sequential data tasks. These findings pave the way for high-accuracy, real-time learning on resource-constrained edge devices.