# Towards Exact Gradient-based Training on Analog In-memory Computing

arXiv: https://arxiv.org/abs/2406.12774 · HTML: https://mlsanity.com/paper/2406.12774/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 67 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Lower Bounds for Non-Convex Stochastic Optimization](https://mlsanity.com/paper/1912.02365/) (2019)
- [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://mlsanity.com/paper/2304.01433/) (2023)
- [Better Theory for SGD in the Nonconvex World](https://mlsanity.com/paper/2002.03329/) (2020)
- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](https://mlsanity.com/paper/2104.13386/) (2021)
- [An Improved Analysis of Stochastic Gradient Descent with Momentum](https://mlsanity.com/paper/2007.07989/) (2020)
- [Unified Optimal Analysis of the (Stochastic) Gradient Method](https://mlsanity.com/paper/1907.04232/) (2019)
- [Neuromorphic Silicon Photonic Networks](https://mlsanity.com/paper/1611.02272/) (2016)
- [Training of Physical Neural Networks](https://mlsanity.com/paper/2406.03372/) (2024)
- [Wave Physics as an Analog Recurrent Neural Network](https://mlsanity.com/paper/1904.12831/) (2019)
- [Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators](https://mlsanity.com/paper/2302.08469/) (2023)
- [A Guide Through the Zoo of Biased SGD](https://mlsanity.com/paper/2305.16296/) (2023)
- [A flexible and fast PyTorch toolkit for simulating training and inference on analog crossbar arrays](https://mlsanity.com/paper/2104.02184/) (2021)
- [Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization](https://mlsanity.com/paper/1604.03257/) (2016)
- [Energy-based learning algorithms for analog computing: a comparative study](https://mlsanity.com/paper/2312.15103/) (2023)
- [Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidance](https://mlsanity.com/paper/2305.20057/) (2023)
- [Convergence of a Stochastic Gradient Method with Momentum for Non-Smooth Non-Convex Optimization](https://mlsanity.com/paper/2002.05466/) (2020)
- [Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices](https://mlsanity.com/paper/1603.07341/) (2016)
- [On the Influence of Momentum Acceleration on Online Learning](https://mlsanity.com/paper/1603.04136/) (2016)
- [Algorithm for Training Neural Networks on Resistive Device Arrays](https://mlsanity.com/paper/1909.07908/) (2019)
- [Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices](https://mlsanity.com/paper/1705.08014/) (2017)
- [AnalogNAS: A Neural Network Design Framework for Accurate Inference with Analog In-Memory Computing](https://mlsanity.com/paper/2305.10459/) (2023)
- [Neural Network Training with Asymmetric Crosspoint Elements](https://mlsanity.com/paper/2201.13377/) (2022)

## Built on by (most cited first, 7 of 7)

- [Analog In-memory Training on General Non-ideal Resistive Elements: The Impact of Response Functions](https://mlsanity.com/paper/2502.06309/) (2025)
- [In-memory Training on Analog Devices with Limited Conductance States via Multi-tile Residual Learning](https://mlsanity.com/paper/2510.02516/) (2025)
- [Fully analogue in-memory neural computing via quantum tunneling effect](https://mlsanity.com/paper/2510.23638/) (2025)
- [Dynamic Symmetric Point Tracking: Tackling Non-ideal Reference in Analog In-memory Training](https://mlsanity.com/paper/2602.21321/) (2026)
- [Making Analog Training Scale: Co-Designing Mapping, Optimizer, and Converters](https://mlsanity.com/paper/2609.36584/) (2026)
- [On the Convergence Theory of Pipeline Gradient-based Analog In-memory Training](https://mlsanity.com/paper/2410.15155/) (2024)
- [Robust Parameter-Efficient LLM Adaptation on Analog Hardware](https://mlsanity.com/paper/2610.05318/) (2026)

## Abstract (arXiv)

Given the high economic and environmental costs of using large vision or language models, analog in-memory accelerators present a promising solution for energy-efficient AI. While inference on analog accelerators has been studied recently, the training perspective is underexplored. Recent studies have shown that the "workhorse" of digital AI training - stochastic gradient descent (SGD) algorithm converges inexactly when applied to model training on non-ideal devices. This paper puts forth a theoretical foundation for gradient-based training on analog devices. We begin by characterizing the non-convergent issue of SGD, which is caused by the asymmetric updates on the analog devices. We then provide a lower bound of the asymptotic error to show that there is a fundamental performance limit of SGD-based analog training rather than an artifact of our analysis. To address this issue, we study a heuristic analog algorithm called Tiki-Taka that has recently exhibited superior empirical performance compared to SGD and rigorously show its ability to exactly converge to a critical point and hence eliminates the asymptotic error. The simulations verify the correctness of the analyses.