# Convergence, design and training of continuous-time dropout as a random batch method

arXiv: https://arxiv.org/abs/2510.13134 · HTML: https://mlsanity.com/paper/2510.13134/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 44 of its 72 references that are in the map.
- First posted on arXiv 15 Oct 2025.

## Builds on (most cited first, 30 of 44)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [Dynamic Network Surgery for Efficient DNNs](https://mlsanity.com/paper/1608.04493/) (2016)
- [Efficient Object Localization Using Convolutional Networks](https://mlsanity.com/paper/1411.4280/) (2014)
- [Concrete Dropout](https://mlsanity.com/paper/1705.07832/) (2017)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Neural Jump Stochastic Differential Equations](https://mlsanity.com/paper/1905.10403/) (2019)
- [Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping](https://mlsanity.com/paper/2010.13369/) (2020)
- [Maximum Principle Based Algorithms for Deep Learning](https://mlsanity.com/paper/1710.09513/) (2017)
- [A Mean-Field Optimal Control Formulation of Deep Learning](https://mlsanity.com/paper/1807.01083/) (2018)
- [Error bounds, quadratic growth, and linear convergence of proximal methods](https://mlsanity.com/paper/1602.06661/) (2016)
- [STEER: Simple Temporal Regularization For Neural ODEs](https://mlsanity.com/paper/2006.10711/) (2020)
- [Structured Bayesian Pruning via Log-Normal Multiplicative Noise](https://mlsanity.com/paper/1705.07283/) (2017)
- [Deep neural networks, generic universal interpolation, and controlled ODEs](https://mlsanity.com/paper/1908.07838/) (2019)
- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)

## Abstract (arXiv)

We study continuous-time dropout in controlled differential equations. We introduce a random-batch approximation of additive vector fields. On each time interval of length h, a random subset of components is activated and rescaled by its inclusion probabilities, yielding an unbiased approximation of the full field. We prove a uniform-in-time mean-square trajectory error of order mathcal O(h). At the distribution level, we derive Wasserstein and pointwise density estimates, together with global L1 bounds under moment assumptions. For supervised training, we prove uniform mathcal O(√ h) root-mean-square fluctuations of the randomized objective and an mathcal O(h) weak error for its expectation, leading to consistency of optimal values and near-minimizers and, under quadratic growth, convergence in control space. We further characterize how the sampling law affects the error constant through inclusion probabilities and field variance, compare fixed-size sampling, fixed partitions, and Bernoulli dropout, and derive a work--accuracy model for selecting the switching scale. Numerical experiments with neural ODEs illustrate trajectory and transport errors, sampling effects, and objective consistency at a fixed control.