Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation
Not on the map. Why?
Abstract (arXiv)
Deep ensembles provide the most reliable uncertainty estimates in deep learning, but their cost grows linearly with the number of members. Implicit ensembles lower this cost by sharing a single backbone across members. Member diversity is a primary determinant of ensemble quality, yet no implicit ensemble can shape it during training; existing methods fix it at initialisation or build it into the architecture. We introduce σN-Ens, a normalisation-based implicit ensemble that treats each member as a task in a multi-task architecture and modulates the shared backbone through sigmoid-bounded scalers. We also introduce a softmax-temperature regulariser, which shapes the equilibrium level of sharing between members and traces the accuracy-calibration frontier. Because only normalisation layers are replicated, the mechanism can wrap convolutional and transformer backbones alike, also allowing pretrained models to be adapted through a short fine-tune. We frame the epistemic uncertainty such an ensemble expresses as modulation uncertainty, and explain why its calibration holds under input corruption, and why its out-of-distribution detection is weaker. Our method is evaluated across ResNets and transformers on CIFAR-10/100, ImageNet and SST-2. σN-Ens matches or outperforms deep ensembles at a fraction of their parameter cost, scales with ensemble size where partitioning methods collapse, and maintains calibration under distribution shift.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 54 of its 62 references that are in the map. First posted on arXiv 26 Jul 2026.
Builds on
54 of 62 references.
References · 54
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2016Wide Residual Networkscited by 3,070
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
38 more
- 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
- 2018DARTS: Differentiable Architecture Searchcited by 1,798
- 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
- 2020Gradient Surgery for Multi-Task Learningcited by 1,256
- 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
- 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 732
- 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
- 2015Weight Uncertainty in Neural Networkscited by 554
- 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
- 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
- 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
- 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
- 2019Improving Adversarial Robustness via Promoting Ensemble Diversitycited by 213
- 2018Understanding Measures of Uncertainty for Adversarial Example Detectioncited by 177
- 2020Training independent subnetworks for robust predictioncited by 177
- 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
- 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 160
- 2021Epistemic Neural Networkscited by 120
- 2021Repulsive Deep Ensembles are Bayesiancited by 104
- 2020Masksembles for Uncertainty Estimationcited by 88
- 2020Bayesian Deep Learning via Subnetwork Inferencecited by 87
- 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 81
- 2024Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Taskscited by 78
- 2023A Unified Theory of Diversity in Ensemble Learningcited by 71
- 2022Deep Ensembles Work, But Are They Necessary?cited by 63
- 2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
- 2021DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimationcited by 48
- 2024Rethinking Aleatoric and Epistemic Uncertaintycited by 42
- 2020Diversity inducing Information Bottleneck in Model Ensemblescited by 27
- 2022FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulationcited by 24
- 2024LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networkscited by 20
- 2023Pathologies of Predictive Diversity in Deep Ensemblescited by 16
- 2023Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Modelscited by 15
- 2018Uncertainty Estimation via Stochastic Batch Normalizationcited by 11
- 2022Improving robustness and calibration in ensembles with diversity regularizationcited by 8
- 2022Tackling covariate shift with node-based Bayesian neural networkscited by 8
- 2024Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensemblescited by 8
- 2025Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformerscited by 5
50 most cited of 54 All 54 (JSON)
Built on by
Not cited yet.