# Swapout: Learning an ensemble of deep architectures

arXiv: https://arxiv.org/abs/1605.06465 · HTML: https://mlsanity.com/paper/1605.06465/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2016.

## Built on by (most cited first, 19 of 19)

- [Ensemble deep learning: A review](https://mlsanity.com/paper/2104.02395/) (2021)
- [Diverse Weight Averaging for Out-of-Distribution Generalization](https://mlsanity.com/paper/2205.09739/) (2022)
- [MoViNets: Mobile Video Networks for Efficient Video Recognition](https://mlsanity.com/paper/2103.11511/) (2021)
- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [DART: Diversify-Aggregate-Repeat Training Improves Generalization of Neural Networks](https://mlsanity.com/paper/2302.14685/) (2023)
- [Improving robustness and calibration in ensembles with diversity regularization](https://mlsanity.com/paper/2201.10908/) (2022)
- [Self-Ensembling GAN for Cross-Domain Semantic Segmentation](https://mlsanity.com/paper/2112.07999/) (2021)
- [Sequential Bayesian Neural Subnetwork Ensembles](https://mlsanity.com/paper/2206.00794/) (2022)
- [Population-Based Evolutionary Gaming for Unsupervised Person Re-identification](https://mlsanity.com/paper/2306.05236/) (2023)
- [Robust Self-Ensembling Network for Hyperspectral Image Classification](https://mlsanity.com/paper/2104.03765/) (2021)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [Machine and Deep Learning for IoT Security and Privacy: Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2210.13547/) (2022)
- [Model and Data Agreement for Learning with Noisy Labels](https://mlsanity.com/paper/2212.01054/) (2022)
- [Deep Neural Networks Inspired by Differential Equations](https://mlsanity.com/paper/2510.09685/) (2025)
- [Zero Training Overhead Portfolios for Learning to Solve Combinatorial Problems](https://mlsanity.com/paper/2102.03002/) (2021)
- [K-XLNet: A General Method for Combining Explicit Knowledge with Language Model Pretraining](https://mlsanity.com/paper/2104.10649/) (2021)
- [Unified Probabilistic Neural Architecture and Weight Ensembling Improves Model Robustness](https://mlsanity.com/paper/2210.04083/) (2022)
- [Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets](https://mlsanity.com/paper/2607.16761/) (2026)
- [Random Recursive Models](https://mlsanity.com/paper/2610.00541/) (2026)

## Abstract (arXiv)

We describe Swapout, a new stochastic training method, that outperforms ResNets of identical network structure yielding impressive results on CIFAR-10 and CIFAR-100. Swapout samples from a rich set of architectures including dropout, stochastic depth and residual architectures as special cases. When viewed as a regularization method swapout not only inhibits co-adaptation of units in a layer, similar to dropout, but also across network layers. We conjecture that swapout achieves strong regularization by implicitly tying the parameters across layers. When viewed as an ensemble training method, it samples a much richer set of architectures than existing methods such as dropout or stochastic depth. We propose a parameterization that reveals connections to exiting architectures and suggests a much richer set of architectures to be explored. We show that our formulation suggests an efficient training method and validate our conclusions on CIFAR-10 and CIFAR-100 matching state of the art accuracy. Remarkably, our 32 layer wider model performs similar to a 1001 layer ResNet model.