Data to 7 Oct 2026

BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning

arXiv 2002.06715 ↗First posted 17 Feb 2020cs.LG stat.MLcited by 368

Not on the map. Why?

Abstract (arXiv)

Ensembles, where multiple neural networks are trained individually and their predictions are averaged, have been shown to be widely successful for improving both the accuracy and predictive uncertainty of single neural networks. However, an ensemble's cost for both training and testing increases linearly with the number of networks, which quickly becomes untenable. In this paper, we propose BatchEnsemble, an ensemble method whose computational and memory costs are significantly lower than typical ensembles. BatchEnsemble achieves this by defining each weight matrix to be the Hadamard product of a shared weight among all ensemble members and a rank-one matrix per member. Unlike ensembles, BatchEnsemble is not only parallelizable across devices, where one device trains one member, but also parallelizable within a device, where multiple ensemble members are updated simultaneously for a given mini-batch. Across CIFAR-10, CIFAR-100, WMT14 EN-DE/EN-FR translation, and out-of-distribution tasks, BatchEnsemble yields competitive accuracy and uncertainties as typical ensembles; the speedup at test time is 3X and memory reduction is 3X at an ensemble of size 4. We also apply BatchEnsemble to lifelong learning, where on Split-CIFAR-100, BatchEnsemble yields comparable performance to progressive neural networks while having a much lower computational and memory costs. We further show that BatchEnsemble can easily scale up to lifelong learning on Split-ImageNet which involves 100 sequential learning tasks.

Key facts. Cited by 368 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 74; 2022: 79; 2023: 80; 2024: 62; 2025: 44; 2026: 29.

368 papers.

Sort:

By area:

2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
2021A Survey of Uncertainty in Deep Neural Networkscited by 675
2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
2021Revisiting the Calibration of Modern Neural Networkscited by 387
2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 228
2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
2021Deep Learning Through the Lens of Example Difficultycited by 139
2023Non-Parametric Outlier Synthesiscited by 129

38 more

2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
2024TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensemblingcited by 105
2021Repulsive Deep Ensembles are Bayesiancited by 103
2021DEUP: Direct Epistemic Uncertainty Predictioncited by 102
2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
2021Priors in Bayesian Deep Learning: A Reviewcited by 101
2021Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognitioncited by 95
2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 93
2022Deep Isolation Forest for Anomaly Detectioncited by 90
2021Soft Calibration Objectives for Neural Networkscited by 89
2023Deep Model Fusion: A Surveycited by 89
2021Learning Neural Network Subspacescited by 82
2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
2022Parameter-efficient Model Adaptation for Vision Transformerscited by 81
2021Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classificationcited by 79
2024LUQ: Long-text Uncertainty Quantification for LLMscited by 79
2023Dream the Impossible: Outlier Imagination with Diffusion Modelscited by 77
2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
2021Classifier Calibration: A survey on how to assess and improve predicted class probabilitiescited by 71
2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 71
2021Representation Topology Divergence: A Method for Comparing Neural Network Representationscited by 66
2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
2023LoRA ensembles for large language model fine-tuningcited by 65
2022Test-Time Adaptation via Self-Training with Nearest Neighbor Informationcited by 64
2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
2022Deep Ensembles Work, But Are They Necessary?cited by 63
2021On the Practicality of Deterministic Epistemic Uncertaintycited by 62
2021No One Representation to Rule Them All: Overlapping Features of Training Methodscited by 57
2024A Survey of Uncertainty Estimation in LLMs: Theory Meets Practicecited by 54
2021Model Zoo: A Growing "Brain" That Learns Continuallycited by 53
2022Assaying Out-Of-Distribution Generalization in Transfer Learningcited by 53
2021Efficient Feature Transformations for Discriminative and Generative Continual Learningcited by 50
2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
2023Anti-Exploration by Random Network Distillationcited by 49

50 most cited of 368 All 368 (JSON)