# First-Passage Approach to Optimizing Perturbations for Improved Training of Machine Learning Models

arXiv: https://arxiv.org/abs/2502.04121 · HTML: https://mlsanity.com/paper/2502.04121/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 64 references that are in the map.
- First posted on arXiv 6 Feb 2025.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Skillful Precipitation Nowcasting using Deep Generative Models of Radar](https://mlsanity.com/paper/2104.00954/) (2021)
- [Stochastic Gradient Descent as Approximate Bayesian Inference](https://mlsanity.com/paper/1704.04289/) (2017)
- [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014)
- [FFCV: Accelerating Training by Removing Data Bottlenecks](https://mlsanity.com/paper/2306.12517/) (2023)
- [Neural Thermodynamic Integration: Free Energies from Energy-based Diffusion Models](https://mlsanity.com/paper/2406.02313/) (2024)
- [When Does Re-initialization Work?](https://mlsanity.com/paper/2206.10011/) (2022)
- [Accelerating Minibatch Stochastic Gradient Descent using Typicality Sampling](https://mlsanity.com/paper/1903.04192/) (2019)
- [TherML: Thermodynamics of Machine Learning](https://mlsanity.com/paper/1807.04162/) (2018)
- [Partitioned integrators for thermodynamic parameterization of neural networks](https://mlsanity.com/paper/1908.11843/) (2019)
- [Charting the Topography of the Neural Network Landscape with Thermal-Like Noise](https://mlsanity.com/paper/2304.01335/) (2023)
- [Stochastic Resetting Mitigates Latent Gradient Bias of SGD from Label Noise](https://mlsanity.com/paper/2406.00396/) (2024)
- [PR-DAD: Phase Retrieval Using Deep Auto-Decoders](https://mlsanity.com/paper/2204.09051/) (2022)

## Built on by (most cited first, 3 of 3)

- [More Bang for the Buck: Improving the Inference of Large Language Models at a Fixed Budget using Reset and Discard (ReD)](https://mlsanity.com/paper/2601.21522/) (2026)
- [Stochastic Resetting Accelerates Reinforcement Learning Beyond Random Search](https://mlsanity.com/paper/2603.16842/) (2026)
- [Ridge Regression from Poisson Resetting: A Renewal Perspective on Spectral Regularization](https://mlsanity.com/paper/2605.30059/) (2026)

## Abstract (arXiv)

Machine learning models have become indispensable tools in applications across the physical sciences. Their training is often time-consuming, vastly exceeding the inference timescales. Several protocols have been developed to perturb the learning process and improve the training, such as shrink and perturb, warm restarts, and stochastic resetting. For classifiers, these perturbations have been shown to result in enhanced speedups or improved generalization. However, the design of such perturbations is usually done ad hoc by intuition and trial and error. To rationally optimize training protocols, we frame them as first-passage processes and consider their response to perturbations. We show that if the unperturbed learning process reaches a quasi-steady state, the response at a single perturbation frequency can predict the behavior at a wide range of frequencies. We employ this approach to a CIFAR-10 classifier using the ResNet-18 model and identify a useful perturbation and frequency among several possibilities. We demonstrate the transferability of the approach to other datasets, architectures, optimizers and even tasks (regression instead of classification). Our work allows optimization of perturbations for improving the training of machine learning models using a first-passage approach.