# Minibatch training of neural network ensembles via trajectory sampling

arXiv: https://arxiv.org/abs/2306.13442 · HTML: https://mlsanity.com/paper/2306.13442/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 23 Jun 2023.

## Builds on (most cited first, 19 of 19)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://mlsanity.com/paper/2304.01852/) (2023)
- [Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach](https://mlsanity.com/paper/1805.00915/) (2018)
- [On Markov chain Monte Carlo methods for tall data](https://mlsanity.com/paper/1505.02827/) (2015)
- [Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks](https://mlsanity.com/paper/2202.00293/) (2022)
- [Interference and Generalization in Temporal Difference Learning](https://mlsanity.com/paper/2003.06350/) (2020)
- [Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models](https://mlsanity.com/paper/2012.01988/) (2020)
- [Austerity in MCMC Land: Cutting the Metropolis-Hastings Budget](https://mlsanity.com/paper/1304.5299/) (2013)
- [Effective Extensible Programming: Unleashing Julia on GPUs](https://mlsanity.com/paper/1712.03112/) (2017)
- [An Efficient Minibatch Acceptance Test for Metropolis-Hastings](https://mlsanity.com/paper/1610.06848/) (2016)
- [Correspondence between neuroevolution and gradient descent](https://mlsanity.com/paper/2008.06643/) (2020)
- [Efficiency of quantum versus classical annealing in non-convex learning problems](https://mlsanity.com/paper/1706.08470/) (2017)
- [Machine learning in and out of equilibrium](https://mlsanity.com/paper/2306.03521/) (2023)
- [Training neural network ensembles via trajectory sampling](https://mlsanity.com/paper/2209.11116/) (2022)

## Abstract (arXiv)

Most iterative neural network training methods use estimates of the loss function over small random subsets (or minibatches) of the data to update the parameters, which aid in decoupling the training time from the (often very large) size of the training datasets. Here, we show that a minibatch approach can also be used to train neural network ensembles (NNEs) via trajectory methods in a highly efficient manner. We illustrate this approach by training NNEs to classify images in the MNIST datasets. This method gives an improvement to the training times, allowing it to scale as the ratio of the size of the dataset to that of the average minibatch size which, in the case of MNIST, gives a computational improvement typically of two orders of magnitude. We highlight the advantage of using longer trajectories to represent NNEs, both for improved accuracy in inference and reduced update cost in terms of the samples needed in minibatch updates.