# Energy Efficient Learning with Low Resolution Stochastic Domain Wall Synapse Based Deep Neural Networks

arXiv: https://arxiv.org/abs/2111.07284 · HTML: https://mlsanity.com/paper/2111.07284/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 54 references that are in the map.
- First posted on arXiv 14 Nov 2021.

## Builds on (most cited first, 8 of 8)

- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Convolutional Neural Networks using Logarithmic Data Representation](https://mlsanity.com/paper/1603.01025/) (2016)
- [Dark Memory and Accelerator-Rich System Optimization in the Dark Silicon Era](https://mlsanity.com/paper/1602.04183/) (2016)
- [On-chip learning for domain wall synapse based Fully Connected Neural Network](https://mlsanity.com/paper/1811.09966/) (2018)
- [Comparing domain wall synapse with other Non Volatile Memory devices for on-chip learning in Analog Hardware Neural Network](https://mlsanity.com/paper/1910.12919/) (2019)

## Built on by (most cited first, 3 of 3)

- [Spintronic Physical Reservoir for Autonomous Prediction and Long-Term Household Energy Load Forecasting](https://mlsanity.com/paper/2304.03343/) (2023)
- [Machine learning using magnetic stochastic synapses](https://mlsanity.com/paper/2303.01886/) (2023)
- [Quantized Non-Volatile Nanomagnetic Synapse based Autoencoder for Efficient Unsupervised Network Anomaly Detection](https://mlsanity.com/paper/2309.06449/) (2023)

## Abstract (arXiv)

We demonstrate that extremely low resolution quantized (nominally 5-state) synapses with large stochastic variations in Domain Wall (DW) position can be both energy efficient and achieve reasonably high testing accuracies compared to Deep Neural Networks (DNNs) of similar sizes using floating precision synaptic weights. Specifically, voltage controlled DW devices demonstrate stochastic behavior as modeled rigorously with micromagnetic simulations and can only encode limited states; however, they can be extremely energy efficient during both training and inference. We show that by implementing suitable modifications to the learning algorithms, we can address the stochastic behavior as well as mitigate the effect of their low-resolution to achieve high testing accuracies. In this study, we propose both in-situ and ex-situ training algorithms, based on modification of the algorithm proposed by Hubara et al. [1] which works well with quantization of synaptic weights. We train several 5-layer DNNs on MNIST dataset using 2-, 3- and 5-state DW device as synapse. For in-situ training, a separate high precision memory unit is adopted to preserve and accumulate the weight gradients, which are then quantized to program the low precision DW devices. Moreover, a sizeable noise tolerance margin is used during the training to address the intrinsic programming noise. For ex-situ training, a precursor DNN is first trained based on the characterized DW device model and a noise tolerance margin, which is similar to the in-situ training. Remarkably, for in-situ inference the energy dissipation to program the devices is only 13 pJ per inference given that the training is performed over the entire MNIST dataset for 10 epochs.