# Spatial-SpinDrop: Spatial Dropout-based Binary Bayesian Neural Network with Spintronics Implementation

arXiv: https://arxiv.org/abs/2306.10185 · HTML: https://mlsanity.com/paper/2306.10185/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 16 Jun 2023.

## Builds on (most cited first, 13 of 13)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [Forward and Backward Information Retention for Accurate Binary Neural Networks](https://mlsanity.com/paper/1909.10788/) (2019)
- [Regularizing Activation Distribution for Training Binarized Deep Networks](https://mlsanity.com/paper/1904.02823/) (2019)
- [Loss-aware Binarization of Deep Networks](https://mlsanity.com/paper/1611.01600/) (2016)
- [VIBNN: Hardware Acceleration of Bayesian Neural Networks](https://mlsanity.com/paper/1802.00822/) (2018)
- [Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices](https://mlsanity.com/paper/1705.08014/) (2017)
- [Efficient Computation Reduction in Bayesian Neural Networks Through Feature Decomposition and Memorization](https://mlsanity.com/paper/2005.03857/) (2020)
- [Bayesian Inference on Binary Spiking Networks Leveraging Nanoscale Device Stochasticity](https://mlsanity.com/paper/2302.01302/) (2023)

## Built on by (most cited first, 7 of 7)

- [Scale-Dropout: Estimating Uncertainty in Deep Neural Networks Using Stochastic Scale](https://mlsanity.com/paper/2311.15816/) (2023)
- [Tiny Deep Ensemble: Uncertainty Estimation in Edge AI Accelerators via Ensembling Normalization Layers with Shared Weights](https://mlsanity.com/paper/2405.05286/) (2024)
- [Concurrent Self-testing of Neural Networks Using Uncertainty Fingerprint](https://mlsanity.com/paper/2401.01458/) (2024)
- [Testing Spintronics Implemented Monte Carlo Dropout-Based Bayesian Neural Networks](https://mlsanity.com/paper/2401.04744/) (2024)
- [NeuSpin: Design of a Reliable Edge Neuromorphic System Based on Spintronics for Green AI](https://mlsanity.com/paper/2401.06195/) (2024)
- [Scalable and Efficient Methods for Uncertainty Estimation and Reduction in Deep Learning](https://mlsanity.com/paper/2401.07145/) (2024)
- [Enhancing Reliability of Neural Networks at the Edge: Inverted Normalization with Stochastic Affine Transformations](https://mlsanity.com/paper/2401.12416/) (2024)

## Abstract (arXiv)

Recently, machine learning systems have gained prominence in real-time, critical decision-making domains, such as autonomous driving and industrial automation. Their implementations should avoid overconfident predictions through uncertainty estimation. Bayesian Neural Networks (BayNNs) are principled methods for estimating predictive uncertainty. However, their computational costs and power consumption hinder their widespread deployment in edge AI. Utilizing Dropout as an approximation of the posterior distribution, binarizing the parameters of BayNNs, and further to that implementing them in spintronics-based computation-in-memory (CiM) hardware arrays provide can be a viable solution. However, designing hardware Dropout modules for convolutional neural network (CNN) topologies is challenging and expensive, as they may require numerous Dropout modules and need to use spatial information to drop certain elements. In this paper, we introduce MC-SpatialDropout, a spatial dropout-based approximate BayNNs with spintronics emerging devices. Our method utilizes the inherent stochasticity of spintronic devices for efficient implementation of the spatial dropout module compared to existing implementations. Furthermore, the number of dropout modules per network layer is reduced by a factor of 9× and energy consumption by a factor of 94.11×, while still achieving comparable predictive performance and uncertainty estimates compared to related works.