# Analog In-memory Training on General Non-ideal Resistive Elements: The Impact of Response Functions

arXiv: https://arxiv.org/abs/2502.06309 · HTML: https://mlsanity.com/paper/2502.06309/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 85 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://mlsanity.com/paper/2304.01433/) (2023)
- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](https://mlsanity.com/paper/2104.13386/) (2021)
- [On Penalty-based Bilevel Gradient Descent Method](https://mlsanity.com/paper/2302.05185/) (2023)
- [Neuromorphic Silicon Photonic Networks](https://mlsanity.com/paper/1611.02272/) (2016)
- [Training End-to-End Analog Neural Networks with Equilibrium Propagation](https://mlsanity.com/paper/2006.01981/) (2020)
- [On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximation](https://mlsanity.com/paper/2309.01753/) (2023)
- [Training of Physical Neural Networks](https://mlsanity.com/paper/2406.03372/) (2024)
- [Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators](https://mlsanity.com/paper/2302.08469/) (2023)
- [Wave Physics as an Analog Recurrent Neural Network](https://mlsanity.com/paper/1904.12831/) (2019)
- [A flexible and fast PyTorch toolkit for simulating training and inference on analog crossbar arrays](https://mlsanity.com/paper/2104.02184/) (2021)
- [Energy-based learning algorithms for analog computing: a comparative study](https://mlsanity.com/paper/2312.15103/) (2023)
- [Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices](https://mlsanity.com/paper/1603.07341/) (2016)
- [Equilibrium Propagation with Continual Weight Updates](https://mlsanity.com/paper/2005.04168/) (2020)
- [Non-Convex Bilevel Games with Critical Point Selection Maps](https://mlsanity.com/paper/2207.04888/) (2022)
- [Algorithm for Training Neural Networks on Resistive Device Arrays](https://mlsanity.com/paper/1909.07908/) (2019)
- [Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices](https://mlsanity.com/paper/1705.08014/) (2017)
- [Fast offset corrected in-memory training](https://mlsanity.com/paper/2303.04721/) (2023)
- [Zero-shifting Technique for Deep Neural Network Training on Resistive Cross-point Arrays](https://mlsanity.com/paper/1907.10228/) (2019)
- [Towards Exact Gradient-based Training on Analog In-memory Computing](https://mlsanity.com/paper/2406.12774/) (2024)
- [PIM-QAT: Neural Network Quantization for Processing-In-Memory (PIM) Systems](https://mlsanity.com/paper/2209.08617/) (2022)

## Built on by (most cited first, 5 of 5)

- [In-memory Training on Analog Devices with Limited Conductance States via Multi-tile Residual Learning](https://mlsanity.com/paper/2510.02516/) (2025)
- [Dynamic Symmetric Point Tracking: Tackling Non-ideal Reference in Analog In-memory Training](https://mlsanity.com/paper/2602.21321/) (2026)
- [Circuit realization and hardware linearization of monotone operator equilibrium networks](https://mlsanity.com/paper/2509.13793/) (2025)
- [Making Analog Training Scale: Co-Designing Mapping, Optimizer, and Converters](https://mlsanity.com/paper/2609.36584/) (2026)
- [Robust Parameter-Efficient LLM Adaptation on Analog Hardware](https://mlsanity.com/paper/2610.05318/) (2026)

## Abstract (arXiv)

As the economic and environmental costs of training and deploying large vision or language models increase dramatically, analog in-memory computing (AIMC) emerges as a promising energy-efficient solution. However, the training perspective, especially its training dynamic, is underexplored. In AIMC hardware, the trainable weights are represented by the conductance of resistive elements and updated using consecutive electrical pulses. While the conductance changes by a constant in response to each pulse, in reality, the change is scaled by asymmetric and non-linear response functions, leading to a non-ideal training dynamic. This paper provides a theoretical foundation for gradient-based training on AIMC hardware with non-ideal response functions. We demonstrate that asymmetric response functions negatively impact Analog SGD by imposing an implicit penalty on the objective. To overcome the issue, we propose Residual Learning algorithm, which provably converges exactly to a critical point by solving a bilevel optimization problem. We demonstrate that the proposed method can be extended to address other hardware imperfections, such as limited response granularity. As we know, it is the first paper to investigate the impact of a class of generic non-ideal response functions. The conclusion is supported by simulations validating our theoretical insights.