# Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks

arXiv: https://arxiv.org/abs/2607.08406 · HTML: https://mlsanity.com/paper/2607.08406/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 50 references that are in the map.
- First posted on arXiv 9 Jul 2026.

## Builds on (most cited first, 19 of 19)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Tensorizing Neural Networks](https://mlsanity.com/paper/1509.06569/) (2015)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Unsupervised Generative Modeling Using Matrix Product States](https://mlsanity.com/paper/1709.01662/) (2017)
- [A Survey on Uncertainty Quantification Methods for Deep Learning](https://mlsanity.com/paper/2302.13425/) (2023)
- [High-performance deep spiking neural networks with 0.3 spikes per neuron](https://mlsanity.com/paper/2306.08744/) (2023)
- [Machine learning with tree tensor networks, CP rank constraints, and tensor dropout](https://mlsanity.com/paper/2305.19440/) (2023)
- [Bayesian neural networks via MCMC: a Python-based tutorial](https://mlsanity.com/paper/2304.02595/) (2023)
- [Inferring Global Dynamics Using a Learning Machine](https://mlsanity.com/paper/2009.13032/) (2020)
- [Energy--Information Trade-off Induces Continuous and Discontinuous Phase Transitions in Lateral Predictive Coding](https://mlsanity.com/paper/2302.11681/) (2023)
- [Performance Portable Monte Carlo Particle Transport on Intel, NVIDIA, and AMD GPUs](https://mlsanity.com/paper/2403.12345/) (2024)
- [Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity](https://mlsanity.com/paper/2502.01330/) (2025)
- [A General Theory for Training Learning Machine](https://mlsanity.com/paper/1704.06885/) (2017)
- [Network Dynamics-Based Framework for Understanding Deep Neural Networks](https://mlsanity.com/paper/2501.02436/) (2025)

## Abstract (arXiv)

Backpropagation (BP) dominates deep learning training, but its reliance on gradients brings inherent troubles -- vanishing and exploding gradients. The pursuit of gradient-free methods has long been a goal in the field of artificial intelligence. This paper shows that indeed the simplest Monte Carlo algorithm implemented on a single GPU -- randomly mutate a parameter, keep it if the loss decreases, otherwise retry -- can practically train deep networks. This gradient-free method does not even need common techniques such as batch normalization or residual connections to directly train sufficiently deep networks. More remarkably, its flexibility extends to several nontrivial scenarios: it enables pure pruning training, supports discrete weights, accommodates unconventional transfer functions such as Gaussian, and reveals the substantial redundancy of deep networks. We have demonstrated its feasibility on deep networks with more than 20 layers, single-hidden-layer wide networks with up to 16,384 hidden neurons, and even a simple Transformer architecture trained on both image classification (MNIST) and character-level language modeling (Tiny Shakespeare). This simple gradient-free method may offer a complementary perspective for understanding the self-organization and learning mechanisms of neural networks, and also provides an alternative route for building physically inspired deep learning systems.