mlsanityData to 7 Oct 2026

Scaling Forward Gradient With Local Losses

arXiv 2210.03310 ↗First posted 7 Oct 2022cs.LG cs.CV cs.NEcited by 65

Not on the map. Why?

Abstract (arXiv)

Forward gradient learning computes a noisy directional gradient and is a biologically plausible alternative to backprop for learning deep neural networks. However, the standard forward gradient algorithm, when applied naively, suffers from high variance when the number of parameters to be learned is large. In this paper, we propose a series of architectural and algorithmic modifications that together make forward gradient learning practical for standard deep learning benchmark tasks. We show that it is possible to substantially reduce the variance of the forward gradient estimator by applying perturbations to activations rather than weights. We further improve the scalability of forward gradient by introducing a large number of local greedy loss functions, each of which involves only a small number of learnable parameters, and a new MLPMixer-inspired architecture, LocalMixer, that is more suitable for local learning. Our approach matches backprop on MNIST and CIFAR-10 and significantly outperforms previously proposed backprop-free algorithms on ImageNet.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 37 of its 79 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

37 of 79 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2016Layer Normalizationcited by 5,579
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  10. 2018Group Normalizationcited by 1,514
  11. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  12. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
25 more
  1. 2015Semi-Supervised Learning with Ladder Networkscited by 246
  2. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  3. 2014Difference Target Propagationcited by 144
  4. 2019Training Neural Networks with Local Error Signalscited by 135
  5. 2018Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batchescited by 133
  6. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 113
  7. 2018Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architecturescited by 100
  8. 2019Deep Learning without Weight Transportcited by 87
  9. 2019Putting An End to End-to-End: Gradient-Isolated Learning of Representationscited by 85
  10. 2014How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagationcited by 77
  11. 2019Decoupled Greedy Learning of CNNscited by 76
  12. 2022Gradients without Backpropagationcited by 73
  13. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  14. 2015How Important is Weight Symmetry in Backpropagation?cited by 57
  15. 2020LoCo: Local Contrastive Representation Learningcited by 55
  16. 2018Biologically-plausible learning algorithms can scale to large datasetscited by 37
  17. 2016Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groupscited by 32
  18. 2019Gated Linear Networkscited by 26
  19. 2021Credit Assignment Through Broadcasting a Global Error Vectorcited by 19
  20. 2020Parallel Training of Deep Networks with Local Updatescited by 18
  21. 2020Interlocking Backpropagation: Improving depthwise model-parallelismcited by 17
  22. 2016Normalizing the Normalizers: Comparing and Extending Network Normalization Schemescited by 12
  23. 2015A Theory of Local Learning, the Learning Channel, and the Optimality of Backpropagationcited by 11
  24. 2015STDP as presynaptic activity times rate of change of postsynaptic activitycited by 8
  25. 2017Online Learning with Gated Linear Networkscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 13; 2024: 22; 2025: 16; 2026: 12.

65 papers.

Sort:

By area:

  1. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  2. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  3. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  4. 2024Training of Physical Neural Networkscited by 47
  5. 2023Can Forward Gradient Match Backpropagation?cited by 26
  6. 2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 25
  7. 2023Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignmentcited by 13
  8. 2023Tensor-Compressed Back-Propagation-Free Training for (Physics-Informed) Neural Networkscited by 12
  9. 2024Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimizationcited by 11
  10. 2024Momentum Auxiliary Network for Supervised Local Learningcited by 9
  11. 2025ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Thinkcited by 9
  12. 2023Multiplexed gradient descent: Fast online training of modern datasets on hardware neural networks without backpropagationcited by 8
38 more
  1. 2023Improving equilibrium propagation without weight symmetry through Jacobian homeostasiscited by 8
  2. 2024Scaling Supervised Local Learning with Augmented Auxiliary Networkscited by 8
  3. 2025NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagationcited by 8
  4. 2022Low-Variance Forward Gradients using Direct Feedback Alignment and Momentumcited by 7
  5. 2023How to guess a gradientcited by 7
  6. 2024Advancing Supervised Local Learning Beyond Classification with Long-term Feature Bankcited by 6
  7. 2024Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradientscited by 6
  8. 2025Learning long range dependencies through time reversal symmetry breakingcited by 6
  9. 2024MLAAN: Scaling Supervised Local Learning with Multilaminar Leap Augmented Auxiliary Networkcited by 5
  10. 2024HPFF: Hierarchical Locally Supervised Learning with Patch Feature Fusioncited by 5
  11. 2024Second-Order Forward-Mode Automatic Differentiation for Optimizationcited by 5
  12. 2024Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approachcited by 5
  13. 2023The Integrated Forward-Forward Algorithm: Integrating Forward-Forward and Shallow Backpropagation With Local Lossescited by 4
  14. 2024Towards Interpretable Deep Local Learning with Successive Gradient Reconciliationcited by 4
  15. 2024Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagationcited by 4
  16. 2024Stepping Forward on the Last Milecited by 4
  17. 2025TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Modelscited by 4
  18. 2024Towards training digitally-tied analog blocks via hybrid gradient computationcited by 3
  19. 2024FLOPS: Forward Learning with OPtimal Samplingcited by 3
  20. 2025MAN++: Scaling Momentum Auxiliary Network for Supervised Local Learning in Vision Taskscited by 3
  21. 2023Node Perturbation Can Effectively Train Multi-Layer Neural Networkscited by 2
  22. 2024Accelerating Legacy Numerical Solvers by Non-intrusive Gradient-based Meta-solvingcited by 2
  23. 2024Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networkscited by 2
  24. 2024Noise-based Local Learning using Stochastic Magnetic Tunnel Junctionscited by 2
  25. 2025Gradient Prediction with Control Variates in the Cheap-Forward Regimecited by 2
  26. 2025FedMPDD: Communication-Efficient Federated Learning with Privacy Preservation Attributes via Projected Directional Derivativecited by 2
  27. 2023Building artificial neural circuits for domain-general cognition: a primer on brain-inspired systems-level architecturecited by 1
  28. 2023Leader-Follower Neural Networks with Local Error Signals Inspired by Complex Collectivescited by 1
  29. 2023Hebbian learning inspired estimation of the linear regression parameters from queriescited by 1
  30. 2024Moonwalk: Inverse-Forward Differentiationcited by 1
  31. 2024Replacement Learning: Training Vision Tasks with Fewer Learnable Parameterscited by 1
  32. 2024Improving the Convergence Rates of Forward Gradient Descent with Repeated Samplingcited by 1
  33. 2025Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradientscited by 1
  34. 2025More Than Memory Savings: Zeroth-Order Optimization Mitigates Forgetting in Continual Learningcited by 1
  35. 2026Credit Assignment via Neural Manifold Noise Correlationcited by 1
  36. 2026Local learning for stable backpropagation-free neural network training towards physical learningcited by 1
  37. 2026Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpoweredcited by 1
  38. 2026Backpropagation-Free Trunk Training via the Split Forward Gradientscited by 1

50 most cited of 65 All 65 (JSON)