mlsanityData to 7 Oct 2026

Convergence, design and training of continuous-time dropout as a random batch method

arXiv 2510.13134 ↗First posted 15 Oct 2025cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

We study continuous-time dropout in controlled differential equations. We introduce a random-batch approximation of additive vector fields. On each time interval of length h, a random subset of components is activated and rescaled by its inclusion probabilities, yielding an unbiased approximation of the full field. We prove a uniform-in-time mean-square trajectory error of order mathcal O(h). At the distribution level, we derive Wasserstein and pointwise density estimates, together with global L1 bounds under moment assumptions. For supervised training, we prove uniform mathcal O(√ h) root-mean-square fluctuations of the randomized objective and an mathcal O(h) weak error for its expectation, leading to consistency of optimal values and near-minimizers and, under quadratic growth, convergence in control space. We further characterize how the sampling law affects the error constant through inclusion probabilities and field variance, compare fixed-size sampling, fixed partitions, and Bernoulli dropout, and derive a work--accuracy model for selecting the switching scale. Numerical experiments with neural ODEs illustrate trajectory and transport errors, sampling effects, and objective consistency at a fixed control.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 44 of its 72 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

44 of 72 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2022Flow Matching for Generative Modelingcited by 5,426
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2018Neural Ordinary Differential Equationscited by 3,733
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2015Variational Inference with Normalizing Flowscited by 2,161
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  12. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
32 more
  1. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  2. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  3. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  4. 2017Stable Architectures for Deep Neural Networkscited by 340
  5. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  6. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  7. 2017Concrete Dropoutcited by 227
  8. 2014Efficient Object Localization Using Convolutional Networkscited by 226
  9. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  10. 2019Neural Jump Stochastic Differential Equationscited by 147
  11. 2020Accelerating Training of Transformer-Based Language Models with Progressive Layer Droppingcited by 93
  12. 2017Maximum Principle Based Algorithms for Deep Learningcited by 77
  13. 2018A Mean-Field Optimal Control Formulation of Deep Learningcited by 67
  14. 2016Error bounds, quadratic growth, and linear convergence of proximal methodscited by 59
  15. 2020STEER: Simple Temporal Regularization For Neural ODEscited by 58
  16. 2017Structured Bayesian Pruning via Log-Normal Multiplicative Noisecited by 39
  17. 2019Deep neural networks, generic universal interpolation, and controlled ODEscited by 28
  18. 2022Implicit regularization of dropoutcited by 24
  19. 2020Notes on the Behavior of MC Dropoutcited by 21
  20. 2025Comprehensive Review of Neural Differential Equations for Time Series Analysiscited by 20
  21. 2020On Convergence and Generalization of Dropout Trainingcited by 19
  22. 2020Large-time asymptotics in deep learningcited by 14
  23. 2021Sparse Flows: Pruning Continuous-depth Modelscited by 13
  24. 2018Stochastic Training of Residual Networks: a Differential Equation Viewpointcited by 12
  25. 2022Turnpike in optimal control of PDEs, ResNets, and beyondcited by 10
  26. 2024Constructive approximate transport maps with normalizing flowscited by 4
  27. 2022Stability and Convergence of a Randomized Model Predictive Control Strategycited by 3
  28. 2025Phase Diagram of Dropout for Two-Layer Neural Networks in the Mean-Field Regimecited by 3
  29. 2021A framework for randomized time-splitting in linear-quadratic optimal controlcited by 2
  30. 2023Control in finite and infinite dimensioncited by 1
  31. 2024Mini-batch descent in semiflowscited by 1
  32. 2025Random Batch Methods for Discretized PDEs on Graphscited by 1

Built on by

Not cited yet.