# Scale-Dropout: Estimating Uncertainty in Deep Neural Networks Using Stochastic Scale

arXiv: https://arxiv.org/abs/2311.15816 · HTML: https://mlsanity.com/paper/2311.15816/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 81 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [Efficient Object Localization Using Convolutional Networks](https://mlsanity.com/paper/1411.4280/) (2014)
- [Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding](https://mlsanity.com/paper/1511.02680/) (2015)
- [Forward and Backward Information Retention for Accurate Binary Neural Networks](https://mlsanity.com/paper/1909.10788/) (2019)
- [XNOR-Net++: Improved Binary Neural Networks](https://mlsanity.com/paper/1909.13863/) (2019)
- [DropConnect Is Effective in Modeling Uncertainty of Bayesian Deep Networks](https://mlsanity.com/paper/1906.04569/) (2019)
- [A Survey on Uncertainty Quantification Methods for Deep Learning](https://mlsanity.com/paper/2302.13425/) (2023)
- [Regularizing Activation Distribution for Training Binarized Deep Networks](https://mlsanity.com/paper/1904.02823/) (2019)
- [Loss-aware Binarization of Deep Networks](https://mlsanity.com/paper/1611.01600/) (2016)
- [VIBNN: Hardware Acceleration of Bayesian Neural Networks](https://mlsanity.com/paper/1802.00822/) (2018)
- [Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices](https://mlsanity.com/paper/1705.08014/) (2017)
- [Efficient Computation Reduction in Bayesian Neural Networks Through Feature Decomposition and Memorization](https://mlsanity.com/paper/2005.03857/) (2020)
- [High-Performance FPGA-based Accelerator for Bayesian Neural Networks](https://mlsanity.com/paper/2105.09163/) (2021)
- [Spatial-SpinDrop: Spatial Dropout-based Binary Bayesian Neural Network with Spintronics Implementation](https://mlsanity.com/paper/2306.10185/) (2023)

## Built on by (most cited first, 6 of 6)

- [Enhancing Fairness and Performance in Machine Learning Models: A Multi-Task Learning Approach with Monte-Carlo Dropout and Pareto Optimality](https://mlsanity.com/paper/2404.08230/) (2024)
- [Tiny Deep Ensemble: Uncertainty Estimation in Edge AI Accelerators via Ensembling Normalization Layers with Shared Weights](https://mlsanity.com/paper/2405.05286/) (2024)
- [Testing Spintronics Implemented Monte Carlo Dropout-Based Bayesian Neural Networks](https://mlsanity.com/paper/2401.04744/) (2024)
- [NeuSpin: Design of a Reliable Edge Neuromorphic System Based on Spintronics for Green AI](https://mlsanity.com/paper/2401.06195/) (2024)
- [Scalable and Efficient Methods for Uncertainty Estimation and Reduction in Deep Learning](https://mlsanity.com/paper/2401.07145/) (2024)
- [Active Inference for Micro-Gesture Recognition: EFE-Guided Temporal Sampling and Adaptive Learning](https://mlsanity.com/paper/2603.07559/) (2026)

## Abstract (arXiv)

Uncertainty estimation in Neural Networks (NNs) is vital in improving reliability and confidence in predictions, particularly in safety-critical applications. Bayesian Neural Networks (BayNNs) with Dropout as an approximation offer a systematic approach to quantifying uncertainty, but they inherently suffer from high hardware overhead in terms of power, memory, and computation. Thus, the applicability of BayNNs to edge devices with limited resources or to high-performance applications is challenging. Some of the inherent costs of BayNNs can be reduced by accelerating them in hardware on a Computation-In-Memory (CIM) architecture with spintronic memories and binarizing their parameters. However, numerous stochastic units are required to implement conventional dropout-based BayNN. In this paper, we propose the Scale Dropout, a novel regularization technique for Binary Neural Networks (BNNs), and Monte Carlo-Scale Dropout (MC-Scale Dropout)-based BayNNs for efficient uncertainty estimation. Our approach requires only one stochastic unit for the entire model, irrespective of the model size, leading to a highly scalable Bayesian NN. Furthermore, we introduce a novel Spintronic memory-based CIM architecture for the proposed BayNN that achieves more than 100× energy savings compared to the state-of-the-art. We validated our method to show up to a 1% improvement in predictive performance and superior uncertainty estimates compared to related works.