mlsanityData to 7 Oct 2026

Forward Only Learning for Orthogonal Neural Networks of any Depth

arXiv 2512.20668 ↗First posted 19 Dec 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Backpropagation is still the de facto algorithm used today to train neural networks. With the exponential growth of recent architectures, the computational cost of this algorithm also becomes a burden. The recent PEPITA and forward-only frameworks have proposed promising alternatives, but they failed to scale up to a handful of hidden layers, yet limiting their use. In this paper, we first analyze theoretically the main limitations of these approaches. It allows us the design of a forward-only algorithm, which is equivalent to backpropagation under the linear and orthogonal assumptions. By relaxing the linear assumption, we then introduce FOTON (Forward-Only Training of Orthogonal Networks) that bridges the gap with the backpropagation algorithm. Experimental results show that it outperforms PEPITA, enabling us to train neural networks of any depth, without the need for a backward pass. Moreover its performance on convolutional networks clearly opens up avenues for its application to more advanced architectures. The code is open-sourced at https://github.com/p0lcAi/FOTON .

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 35 references that are in the map. First posted on arXiv 19 Dec 2025.

Builds on

26 of 35 references.

References · 26

  1. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  2. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  3. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  4. 2018Sorting out Lipschitz function approximationcited by 211
  5. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 193
  6. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  7. 2018Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architecturescited by 100
  8. 2019Deep Learning without Weight Transportcited by 87
  9. 2019Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networkscited by 86
  10. 2021Skew Orthogonal Convolutionscited by 67
  11. 2022Scaling Forward Gradient With Local Lossescited by 65
  12. 2022Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Passcited by 63
14 more
  1. 2020Training End-to-End Analog Neural Networks with Equilibrium Propagationcited by 57
  2. 2020Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architecturescited by 56
  3. 2016Regularizing CNNs with Locally Constrained Decorrelationscited by 41
  4. 2018Biologically-plausible learning algorithms can scale to large datasetscited by 37
  5. 2019Learning to solve the credit assignment problemcited by 27
  6. 2019Direct Feedback Alignment with Sparse Connections for Local Learningcited by 26
  7. 2023Can Forward Gradient Match Backpropagation?cited by 26
  8. 2019Principled Training of Neural Networks with Direct Feedback Alignmentcited by 23
  9. 2020Two Routes to Scalable Credit Assignment without Weight Symmetrycited by 19
  10. 2022Spike-based local synaptic plasticity: A survey of computational models and neuromorphic circuitscited by 13
  11. 2023Forward Learning with Top-Down Feedback: Empirical and Analytical Characterizationcited by 13
  12. 2022Optimization without Backpropagationcited by 11
  13. 2019Spike-based causal inference for weight alignmentcited by 10
  14. 2020Asymptotic Singular Value Distribution of Linear Convolutional Layerscited by 2

Built on by

Not cited yet.