# Split-Ensemble: Efficient OOD-aware Ensemble via Task and Model Splitting

arXiv: https://arxiv.org/abs/2312.09148 · HTML: https://mlsanity.com/paper/2312.09148/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Builds on (most cited first, 25 of 25)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [SSD: A Unified Framework for Self-Supervised Outlier Detection](https://mlsanity.com/paper/2103.12051/) (2021)
- [Contrastive Training for Improved Out-of-Distribution Detection](https://mlsanity.com/paper/2007.05566/) (2020)
- [Training independent subnetworks for robust prediction](https://mlsanity.com/paper/2010.06610/) (2020)
- [Semantically Coherent Out-of-Distribution Detection](https://mlsanity.com/paper/2108.11941/) (2021)
- [Masksembles for Uncertainty Estimation](https://mlsanity.com/paper/2012.08334/) (2020)
- [Global Vision Transformer Pruning with Hessian-Aware Saliency](https://mlsanity.com/paper/2110.04869/) (2021)
- [Splitting Steepest Descent for Growing Neural Architectures](https://mlsanity.com/paper/1910.02366/) (2019)
- [Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition](https://mlsanity.com/paper/2207.01160/) (2022)
- [FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation](https://mlsanity.com/paper/2206.00050/) (2022)
- [Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descent](https://mlsanity.com/paper/1910.03103/) (2019)
- [AutoMTL: A Programming Framework for Automating Efficient Multi-Task Learning](https://mlsanity.com/paper/2110.13076/) (2021)
- [Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splitting](https://mlsanity.com/paper/2003.10392/) (2020)
- [DiSparse: Disentangled Sparsification for Multitask Model Compression](https://mlsanity.com/paper/2206.04662/) (2022)

## Built on by (most cited first, 4 of 4)

- [LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks](https://mlsanity.com/paper/2405.14438/) (2024)
- [FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusion](https://mlsanity.com/paper/2406.03280/) (2024)
- [Outlier Synthesis via Hamiltonian Monte Carlo for Out-of-Distribution Detection](https://mlsanity.com/paper/2501.16718/) (2025)
- [Quantifying the Uncertainty of Foundation Models with Singular Value Ensembles](https://mlsanity.com/paper/2601.22068/) (2026)

## Abstract (arXiv)

Uncertainty estimation is crucial for machine learning models to detect out-of-distribution (OOD) inputs. However, the conventional discriminative deep learning classifiers produce uncalibrated closed-set predictions for OOD data. A more robust classifiers with the uncertainty estimation typically require a potentially unavailable OOD dataset for outlier exposure training, or a considerable amount of additional memory and compute to build ensemble models. In this work, we improve on uncertainty estimation without extra OOD data or additional inference costs using an alternative Split-Ensemble method. Specifically, we propose a novel subtask-splitting ensemble training objective, where a common multiclass classification task is split into several complementary subtasks. Then, each subtask's training data can be considered as OOD to the other subtasks. Diverse submodels can therefore be trained on each subtask with OOD-aware objectives. The subtask-splitting objective enables us to share low-level features across submodels to avoid parameter and computational overheads. In particular, we build a tree-like Split-Ensemble architecture by performing iterative splitting and pruning from a shared backbone model, where each branch serves as a submodel corresponding to a subtask. This leads to improved accuracy and uncertainty estimation across submodels under a fixed ensemble computation budget. Empirical study with ResNet-18 backbone shows Split-Ensemble, without additional computation cost, improves accuracy over a single model by 0.8%, 1.8%, and 25.5% on CIFAR-10, CIFAR-100, and Tiny-ImageNet, respectively. OOD detection for the same backbone and in-distribution datasets surpasses a single model baseline by, correspondingly, 2.2%, 8.1%, and 29.6% mean AUROC.