# Neural Network Training with Asymmetric Crosspoint Elements

arXiv: https://arxiv.org/abs/2201.13377 · HTML: https://mlsanity.com/paper/2201.13377/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 39 references that are in the map.
- First posted on arXiv 31 Jan 2022.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices](https://mlsanity.com/paper/1603.07341/) (2016)
- [Algorithm for Training Neural Networks on Resistive Device Arrays](https://mlsanity.com/paper/1909.07908/) (2019)
- [Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices](https://mlsanity.com/paper/1705.08014/) (2017)
- [Zero-shifting Technique for Deep Neural Network Training on Resistive Cross-point Arrays](https://mlsanity.com/paper/1907.10228/) (2019)
- [How neural networks find generalizable solutions: Self-tuned annealing in deep learning](https://mlsanity.com/paper/2001.01678/) (2020)
- [Training LSTM Networks with Resistive Cross-Point Devices](https://mlsanity.com/paper/1806.00166/) (2018)

## Built on by (most cited first, 3 of 3)

- [Fast offset corrected in-memory training](https://mlsanity.com/paper/2303.04721/) (2023)
- [Towards Exact Gradient-based Training on Analog In-memory Computing](https://mlsanity.com/paper/2406.12774/) (2024)
- [On the Convergence Theory of Pipeline Gradient-based Analog In-memory Training](https://mlsanity.com/paper/2410.15155/) (2024)

## Abstract (arXiv)

Analog crossbar arrays comprising programmable nonvolatile resistors are under intense investigation for acceleration of deep neural network training. However, the ubiquitous asymmetric conductance modulation of practical resistive devices critically degrades the classification performance of networks trained with conventional algorithms. Here, we describe and experimentally demonstrate an alternative fully-parallel training algorithm: Stochastic Hamiltonian Descent. Instead of conventionally tuning weights in the direction of the error function gradient, this method programs the network parameters to successfully minimize the total energy (Hamiltonian) of the system that incorporates the effects of device asymmetry. We provide critical intuition on why device asymmetry is fundamentally incompatible with conventional training algorithms and how the new approach exploits it as a useful feature instead. Our technique enables immediate realization of analog deep learning accelerators based on readily available device technologies.