# RSO: A Gradient Free Sampling Based Approach For Training Deep Neural Networks

arXiv: https://arxiv.org/abs/2005.05955 · HTML: https://mlsanity.com/paper/2005.05955/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2020.

## Built on by (most cited first, 5 of 5)

- [Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data](https://mlsanity.com/paper/2408.14874/) (2024)
- [Stochastic Gradient Sampling for Enhancing Neural Networks Training](https://mlsanity.com/paper/2310.17042/) (2023)
- [Training neural networks using Metropolis Monte Carlo and an adaptive variant](https://mlsanity.com/paper/2205.07408/) (2022)
- [Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation](https://mlsanity.com/paper/2308.12053/) (2023)
- [Gradient-Free Training of Quantized Neural Networks](https://mlsanity.com/paper/2410.09734/) (2024)

## Abstract (arXiv)

We propose RSO (random search optimization), a gradient free Markov Chain Monte Carlo search based approach for training deep neural networks. To this end, RSO adds a perturbation to a weight in a deep neural network and tests if it reduces the loss on a mini-batch. If this reduces the loss, the weight is updated, otherwise the existing weight is retained. Surprisingly, we find that repeating this process a few times for each weight is sufficient to train a deep neural network. The number of weight updates for RSO is an order of magnitude lesser when compared to backpropagation with SGD. RSO can make aggressive weight updates in each step as there is no concept of learning rate. The weight update step for individual layers is also not coupled with the magnitude of the loss. RSO is evaluated on classification tasks on MNIST and CIFAR-10 datasets with deep neural networks of 6 to 10 layers where it achieves an accuracy of 99.1% and 81.8% respectively. We also find that after updating the weights just 5 times, the algorithm obtains a classification accuracy of 98% on MNIST.