# Forward Only Learning for Orthogonal Neural Networks of any Depth

arXiv: https://arxiv.org/abs/2512.20668 · HTML: https://mlsanity.com/paper/2512.20668/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 19 Dec 2025.

## Builds on (most cited first, 26 of 26)

- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks](https://mlsanity.com/paper/1802.04034/) (2018)
- [Orthogonalizing Convolutional Layers with the Cayley Transform](https://mlsanity.com/paper/2104.07167/) (2021)
- [Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures](https://mlsanity.com/paper/1807.04587/) (2018)
- [Deep Learning without Weight Transport](https://mlsanity.com/paper/1904.05391/) (2019)
- [Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks](https://mlsanity.com/paper/1911.00937/) (2019)
- [Skew Orthogonal Convolutions](https://mlsanity.com/paper/2105.11417/) (2021)
- [Scaling Forward Gradient With Local Losses](https://mlsanity.com/paper/2210.03310/) (2022)
- [Training End-to-End Analog Neural Networks with Equilibrium Propagation](https://mlsanity.com/paper/2006.01981/) (2020)
- [Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Pass](https://mlsanity.com/paper/2201.11665/) (2022)
- [Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures](https://mlsanity.com/paper/2006.12878/) (2020)
- [Regularizing CNNs with Locally Constrained Decorrelations](https://mlsanity.com/paper/1611.01967/) (2016)
- [Biologically-plausible learning algorithms can scale to large datasets](https://mlsanity.com/paper/1811.03567/) (2018)
- [Learning to solve the credit assignment problem](https://mlsanity.com/paper/1906.00889/) (2019)
- [Direct Feedback Alignment with Sparse Connections for Local Learning](https://mlsanity.com/paper/1903.02083/) (2019)
- [Can Forward Gradient Match Backpropagation?](https://mlsanity.com/paper/2306.06968/) (2023)
- [Principled Training of Neural Networks with Direct Feedback Alignment](https://mlsanity.com/paper/1906.04554/) (2019)
- [Two Routes to Scalable Credit Assignment without Weight Symmetry](https://mlsanity.com/paper/2003.01513/) (2020)
- [Spike-based local synaptic plasticity: A survey of computational models and neuromorphic circuits](https://mlsanity.com/paper/2209.15536/) (2022)
- [Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization](https://mlsanity.com/paper/2302.05440/) (2023)
- [Optimization without Backpropagation](https://mlsanity.com/paper/2209.06302/) (2022)
- [Spike-based causal inference for weight alignment](https://mlsanity.com/paper/1910.01689/) (2019)
- [Asymptotic Singular Value Distribution of Linear Convolutional Layers](https://mlsanity.com/paper/2006.07117/) (2020)

## Abstract (arXiv)

Backpropagation is still the de facto algorithm used today to train neural networks. With the exponential growth of recent architectures, the computational cost of this algorithm also becomes a burden. The recent PEPITA and forward-only frameworks have proposed promising alternatives, but they failed to scale up to a handful of hidden layers, yet limiting their use. In this paper, we first analyze theoretically the main limitations of these approaches. It allows us the design of a forward-only algorithm, which is equivalent to backpropagation under the linear and orthogonal assumptions. By relaxing the linear assumption, we then introduce FOTON (Forward-Only Training of Orthogonal Networks) that bridges the gap with the backpropagation algorithm. Experimental results show that it outperforms PEPITA, enabling us to train neural networks of any depth, without the need for a backward pass. Moreover its performance on convolutional networks clearly opens up avenues for its application to more advanced architectures. The code is open-sourced at https://github.com/p0lcAi/FOTON .