mlsanityData to 14 Oct 2026

Analog In-memory Training on General Non-ideal Resistive Elements: The Impact of Response Functions

arXiv 2502.06309 ↗First posted 10 Feb 2025cs.LG cs.AR math.OCcited by 5

Not on the map. Why?

Abstract (arXiv)

As the economic and environmental costs of training and deploying large vision or language models increase dramatically, analog in-memory computing (AIMC) emerges as a promising energy-efficient solution. However, the training perspective, especially its training dynamic, is underexplored. In AIMC hardware, the trainable weights are represented by the conductance of resistive elements and updated using consecutive electrical pulses. While the conductance changes by a constant in response to each pulse, in reality, the change is scaled by asymmetric and non-linear response functions, leading to a non-ideal training dynamic. This paper provides a theoretical foundation for gradient-based training on AIMC hardware with non-ideal response functions. We demonstrate that asymmetric response functions negatively impact Analog SGD by imposing an implicit penalty on the objective. To overcome the issue, we propose Residual Learning algorithm, which provably converges exactly to a critical point by solving a bilevel optimization problem. We demonstrate that the proposed method can be extended to address other hardware imperfections, such as limited response granularity. As we know, it is the first paper to investigate the impact of a class of generic non-ideal response functions. The conclusion is supported by simulations validating our theoretical insights.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 25 of its 85 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

25 of 85 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  4. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 434
  5. 2016Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagationcited by 259
  6. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  7. 2021Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systemscited by 128
  8. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  9. 2016Neuromorphic Silicon Photonic Networkscited by 68
  10. 2020Training End-to-End Analog Neural Networks with Equilibrium Propagationcited by 63
  11. 2023On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximationcited by 51
  12. 2024Training of Physical Neural Networkscited by 49
13 more
  1. 2019Wave Physics as an Analog Recurrent Neural Networkcited by 43
  2. 2023Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based acceleratorscited by 43
  3. 2021A flexible and fast PyTorch toolkit for simulating training and inference on analog crossbar arrayscited by 39
  4. 2023Energy-based learning algorithms for analog computing: a comparative studycited by 37
  5. 2016Acceleration of Deep Neural Network Training with Resistive Cross-Point Devicescited by 29
  6. 2020Equilibrium Propagation with Continual Weight Updatescited by 26
  7. 2022Non-Convex Bilevel Games with Critical Point Selection Mapscited by 25
  8. 2019Algorithm for Training Neural Networks on Resistive Device Arrayscited by 19
  9. 2017Training Deep Convolutional Neural Networks with Resistive Cross-Point Devicescited by 18
  10. 2023Fast offset corrected in-memory trainingcited by 11
  11. 2019Zero-shifting Technique for Deep Neural Network Training on Resistive Cross-point Arrayscited by 7
  12. 2024Towards Exact Gradient-based Training on Analog In-memory Computingcited by 7
  13. 2022PIM-QAT: Neural Network Quantization for Processing-In-Memory (PIM) Systemscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025In-memory Training on Analog Devices with Limited Conductance States via Multi-tile Residual Learningcited by 3
  2. 2026Dynamic Symmetric Point Tracking: Tackling Non-ideal Reference in Analog In-memory Trainingcited by 2
  3. 2025Circuit realization and hardware linearization of monotone operator equilibrium networkscited by 1
  4. 2026Making Analog Training Scale: Co-Designing Mapping, Optimizer, and Converterscited by 1
  5. 2026Robust Parameter-Efficient LLM Adaptation on Analog Hardwarecited by 0