# One Forward is Enough for Neural Network Training via Likelihood Ratio Method

arXiv: https://arxiv.org/abs/2305.08960 · HTML: https://mlsanity.com/paper/2305.08960/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 38 references that are in the map.
- First posted on arXiv 15 May 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks](https://mlsanity.com/paper/1701.05923/) (2017)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [The Reversible Residual Network: Backpropagation Without Storing Activations](https://mlsanity.com/paper/1707.04585/) (2017)
- [A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1510.03820/) (2015)
- [The HSIC Bottleneck: Deep Learning without Back-Propagation](https://mlsanity.com/paper/1908.01580/) (2019)
- [Deep Residual Correction Network for Partial Domain Adaptation](https://mlsanity.com/paper/2004.04914/) (2020)
- [Noise Optimization for Artificial Neural Networks](https://mlsanity.com/paper/2102.04450/) (2021)

## Built on by (most cited first, 11 of 11)

- [Learning to Transform Dynamically for Better Adversarial Transferability](https://mlsanity.com/paper/2405.14077/) (2024)
- [Discover and Mitigate Multiple Biased Subgroups in Image Classifiers](https://mlsanity.com/paper/2403.12777/) (2024)
- [How to guess a gradient](https://mlsanity.com/paper/2312.04709/) (2023)
- [Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach](https://mlsanity.com/paper/2411.05873/) (2024)
- [Half-order Fine-Tuning for Diffusion Model: A Recursive Likelihood Ratio Optimizer](https://mlsanity.com/paper/2502.00639/) (2025)
- [FLOPS: Forward Learning with OPtimal Sampling](https://mlsanity.com/paper/2410.05966/) (2024)
- [Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networks](https://mlsanity.com/paper/2407.12516/) (2024)
- [Forward Learning for Gradient-based Black-box Saliency Map Generation](https://mlsanity.com/paper/2403.15603/) (2024)
- [Training neural networks without backpropagation using particles](https://mlsanity.com/paper/2412.05667/) (2024)
- [Forward Learning with Differential Privacy](https://mlsanity.com/paper/2504.00411/) (2025)
- [Energy-Efficient Supervised Learning with a Binary Stochastic Forward-Forward Algorithm](https://mlsanity.com/paper/2507.06461/) (2025)

## Abstract (arXiv)

While backpropagation (BP) is the mainstream approach for gradient computation in neural network training, its heavy reliance on the chain rule of differentiation constrains the designing flexibility of network architecture and training pipelines. We avoid the recursive computation in BP and develop a unified likelihood ratio (ULR) method for gradient estimation with just one forward propagation. Not only can ULR be extended to train a wide variety of neural network architectures, but the computation flow in BP can also be rearranged by ULR for better device adaptation. Moreover, we propose several variance reduction techniques to further accelerate the training process. Our experiments offer numerical results across diverse aspects, including various neural network training scenarios, computation flow rearrangement, and fine-tuning of pre-trained models. All findings demonstrate that ULR effectively enhances the flexibility of neural network training by permitting localized module training without compromising the global objective and significantly boosts the network robustness.