# Training neural network ensembles via trajectory sampling

arXiv: https://arxiv.org/abs/2209.11116 · HTML: https://mlsanity.com/paper/2209.11116/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 40 references that are in the map.
- First posted on arXiv 22 Sep 2022.

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)
- [Model-Ensemble Trust-Region Policy Optimization](https://mlsanity.com/paper/1802.10592/) (2018)
- [A high-bias, low-variance introduction to Machine Learning for physicists](https://mlsanity.com/paper/1803.08823/) (2018)
- [Fashionable Modelling with Flux](https://mlsanity.com/paper/1811.01457/) (2018)
- [Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models](https://mlsanity.com/paper/2012.01988/) (2020)
- [Effective Extensible Programming: Unleashing Julia on GPUs](https://mlsanity.com/paper/1712.03112/) (2017)
- [Correspondence between neuroevolution and gradient descent](https://mlsanity.com/paper/2008.06643/) (2020)

## Built on by (most cited first, 2 of 2)

- [Rare Event Analysis of Large Language Models](https://mlsanity.com/paper/2602.06791/) (2026)
- [Minibatch training of neural network ensembles via trajectory sampling](https://mlsanity.com/paper/2306.13442/) (2023)

## Abstract (arXiv)

In machine learning, there is renewed interest in neural network ensembles (NNEs), whereby predictions are obtained as an aggregate from a diverse set of smaller models, rather than from a single larger model. Here, we show how to define and train a NNE using techniques from the study of rare trajectories in stochastic systems. We define an NNE in terms of the trajectory of the model parameters under a simple, and discrete in time, diffusive dynamics, and train the NNE by biasing these trajectories towards a small time-integrated loss, as controlled by appropriate counting fields which act as hyperparameters. We demonstrate the viability of this technique on a range of simple supervised learning tasks. We discuss potential advantages of our trajectory sampling approach compared with more conventional gradient based methods.