# The Trifecta: Three simple techniques for training deeper Forward-Forward networks arXiv: https://arxiv.org/abs/2311.18130 ยท HTML: https://mlsanity.com/paper/2311.18130/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 26 of its 50 references that are in the map. - First posted on arXiv 29 Nov 2023. ## Builds on (most cited first, 26 of 26) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015) - [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016) - [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022) - [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018) - [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017) - [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014) - [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022) - [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015) - [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020) - [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020) - [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017) - [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021) - [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022) - [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016) - [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016) - [Greedy Layerwise Learning Can Scale to ImageNet](https://mlsanity.com/paper/1812.11446/) (2018) - [Putting An End to End-to-End: Gradient-Isolated Learning of Representations](https://mlsanity.com/paper/1905.11786/) (2019) - [Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Pass](https://mlsanity.com/paper/2201.11665/) (2022) - [Random feedback weights support learning in deep neural networks](https://mlsanity.com/paper/1411.0247/) (2014) - [Decoupled Parallel Backpropagation with Convergence Guarantee](https://mlsanity.com/paper/1804.10574/) (2018) - [The Predictive Forward-Forward Algorithm](https://mlsanity.com/paper/2301.01452/) (2023) - [SymBa: Symmetric Backpropagation-Free Contrastive Learning with Forward-Forward Algorithm for Optimizing Convergence](https://mlsanity.com/paper/2303.08418/) (2023) - [Parallel Training of Deep Networks with Local Updates](https://mlsanity.com/paper/2012.03837/) (2020) - [Interlocking Backpropagation: Improving depthwise model-parallelism](https://mlsanity.com/paper/2010.04116/) (2020) ## Built on by (most cited first, 12 of 12) - [Training Convolutional Neural Networks with the Forward-Forward algorithm](https://mlsanity.com/paper/2312.14924/) (2023) - [Distance-Forward Learning: Enhancing the Forward-Forward Algorithm Towards High-Performance On-Chip Learning](https://mlsanity.com/paper/2408.14925/) (2024) - [Hyperspherical Forward-Forward with Prototypical Representations](https://mlsanity.com/paper/2605.00082/) (2026) - [HCL-FF: Hierarchical and Contrastive Learning for Forward-Forward Algorithm](https://mlsanity.com/paper/2605.24797/) (2026) - [Learning Using a Single Forward Pass](https://mlsanity.com/paper/2402.09769/) (2024) - [On Advancements of the Forward-Forward Algorithm](https://mlsanity.com/paper/2504.21662/) (2025) - [Stochastic Layer-wise Learning: Scalable and Efficient Alternative to Backpropagation](https://mlsanity.com/paper/2505.05181/) (2025) - [FFGAF-SNN: The Forward-Forward Based Gradient Approximation Free Training Framework for Spiking Neural Networks](https://mlsanity.com/paper/2507.23643/) (2025) - [Reshaping the Forward-Forward Algorithm with a Similarity-Based Objective](https://mlsanity.com/paper/2509.08697/) (2025) - [Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant](https://mlsanity.com/paper/2605.06240/) (2026) - [FFR: Forward-Forward Learning for Regression](https://mlsanity.com/paper/2606.03927/) (2026) - [Rescuing Points Trapped Inside Incorrect Decision Regions: A target for Evaluating and Improving Forward-Forward Learning](https://mlsanity.com/paper/2610.16347/) (2026) ## Abstract (arXiv) Modern machine learning models are able to outperform humans on a variety of non-trivial tasks. However, as the complexity of the models increases, they consume significant amounts of power and still struggle to generalize effectively to unseen data. Local learning, which focuses on updating subsets of a model's parameters at a time, has emerged as a promising technique to address these issues. Recently, a novel local learning algorithm, called Forward-Forward, has received widespread attention due to its innovative approach to learning. Unfortunately, its application has been limited to smaller datasets due to scalability issues. To this end, we propose The Trifecta, a collection of three simple techniques that synergize exceptionally well and drastically improve the Forward-Forward algorithm on deeper networks. Our experiments demonstrate that our models are on par with similarly structured, backpropagation-based models in both training speed and test accuracy on simple datasets. This is achieved by the ability to learn representations that are informative locally, on a layer-by-layer basis, and retain their informativeness when propagated to deeper layers in the architecture. This leads to around 84% accuracy on CIFAR-10, a notable improvement (25%) over the original FF algorithm. These results highlight the potential of Forward-Forward as a genuine competitor to backpropagation and as a promising research avenue.