# MotherNets: Rapid Deep Ensemble Learning

arXiv: https://arxiv.org/abs/1809.04270 · HTML: https://mlsanity.com/paper/1809.04270/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2018.

## Built on by (most cited first, 14 of 14)

- [MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks](https://mlsanity.com/paper/2103.06132/) (2021)
- [Prune and Tune Ensembles: Low-Cost Ensemble Learning With Sparse Independent Subnetworks](https://mlsanity.com/paper/2202.11782/) (2022)
- [Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Models](https://mlsanity.com/paper/2102.02344/) (2021)
- [Easy Ensemble: Simple Deep Ensemble Learning for Sensor-Based Human Activity Recognition](https://mlsanity.com/paper/2203.04153/) (2022)
- [Improving Accuracy and Explainability of Online Handwriting Recognition](https://mlsanity.com/paper/2209.09102/) (2022)
- [RBUE: A ReLU-Based Uncertainty Estimation Method of Deep Neural Networks](https://mlsanity.com/paper/2107.07197/) (2021)
- [Faster, Lighter, More Accurate: A Deep Learning Ensemble for Content Moderation](https://mlsanity.com/paper/2309.05150/) (2023)
- [Stochastic Subnetwork Annealing: A Regularization Technique for Fine Tuning Pruned Subnetworks](https://mlsanity.com/paper/2401.08830/) (2024)
- [Noisy Deep Ensemble: Accelerating Deep Ensemble Learning via Noise Injection](https://mlsanity.com/paper/2504.05677/) (2025)
- [When Are Two Scores Better Than One? Investigating Ensembles of Diffusion Models](https://mlsanity.com/paper/2601.11444/) (2026)
- [The Case for High-Accuracy Classification: Think Small, Think Many!](https://mlsanity.com/paper/2103.10350/) (2021)
- [On the Effectiveness of Neural Ensembles for Image Classification with Small Datasets](https://mlsanity.com/paper/2111.14493/) (2021)
- [Symbolic Snapshot Ensembles](https://mlsanity.com/paper/2510.24633/) (2025)
- [Sharing Knowledge without Sharing Data: Stitches can improve ensembles of disjointly trained models](https://mlsanity.com/paper/2512.17592/) (2025)

## Abstract (arXiv)

Ensembles of deep neural networks significantly improve generalization accuracy. However, training neural network ensembles requires a large amount of computational resources and time. State-of-the-art approaches either train all networks from scratch leading to prohibitive training cost that allows only very small ensemble sizes in practice, or generate ensembles by training a monolithic architecture, which results in lower model diversity and decreased prediction accuracy. We propose MotherNets to enable higher accuracy and practical training cost for large and diverse neural network ensembles: A MotherNet captures the structural similarity across some or all members of a deep neural network ensemble which allows us to share data movement and computation costs across these networks. We first train a single or a small set of MotherNets and, subsequently, we generate the target ensemble networks by transferring the function from the trained MotherNet(s). Then, we continue to train these ensemble networks, which now converge drastically faster compared to training from scratch. MotherNets handle ensembles with diverse architectures by clustering ensemble networks of similar architecture and training a separate MotherNet for every cluster. MotherNets also use clustering to control the accuracy vs. training cost tradeoff. We show that compared to state-of-the-art approaches such as Snapshot Ensembles, Knowledge Distillation, and TreeNets, MotherNets provide a new Pareto frontier for the accuracy-training cost tradeoff. Crucially, training cost and accuracy improvements continue to scale as we increase the ensemble size (2 to 3 percent reduced absolute test error rate and up to 35 percent faster training compared to Snapshot Ensembles). We verify these benefits over numerous neural network architectures and large data sets.