# Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators

arXiv: https://arxiv.org/abs/2302.08469 · HTML: https://mlsanity.com/paper/2302.08469/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 111 references that are in the map.
- First posted on arXiv 16 Feb 2023.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Maxout Networks](https://mlsanity.com/paper/1302.4389/) (2013)
- [Compute Trends Across Three Eras of Machine Learning](https://mlsanity.com/paper/2202.05924/) (2022)
- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](https://mlsanity.com/paper/2104.13386/) (2021)
- [Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack](https://mlsanity.com/paper/1811.09310/) (2018)
- [Dropout Training as Adaptive Regularization](https://mlsanity.com/paper/1307.1493/) (2013)
- [Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization](https://mlsanity.com/paper/1710.05179/) (2017)
- [A flexible and fast PyTorch toolkit for simulating training and inference on analog crossbar arrays](https://mlsanity.com/paper/2104.02184/) (2021)
- [Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices](https://mlsanity.com/paper/1603.07341/) (2016)
- [RxNN: A Framework for Evaluating Deep Neural Networks on Resistive Crossbars](https://mlsanity.com/paper/1809.00072/) (2018)
- [Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices](https://mlsanity.com/paper/1705.08014/) (2017)
- [Noisy Machines: Understanding Noisy Neural Networks and Enhancing Robustness to Analog Hardware Errors Using Distillation](https://mlsanity.com/paper/2001.04974/) (2020)
- [AI Accelerator Survey and Trends](https://mlsanity.com/paper/2109.08957/) (2021)
- [TxSim:Modeling Training of Deep Neural Networks on Resistive Crossbar Systems](https://mlsanity.com/paper/2002.11151/) (2020)
- [Whiteout: Gaussian Adaptive Noise Regularization in Deep Neural Networks](https://mlsanity.com/paper/1612.01490/) (2016)
- [Embedding-Based Speaker Adaptive Training of Deep Neural Networks](https://mlsanity.com/paper/1710.06937/) (2017)

## Built on by (most cited first, 30 of 43)

- [Training of Physical Neural Networks](https://mlsanity.com/paper/2406.03372/) (2024)
- [Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference](https://mlsanity.com/paper/2406.08413/) (2024)
- [Fast offset corrected in-memory training](https://mlsanity.com/paper/2303.04721/) (2023)
- [Using the IBM Analog In-Memory Hardware Acceleration Kit for Neural Network Training and Inference](https://mlsanity.com/paper/2307.09357/) (2023)
- [AnalogNAS: A Neural Network Design Framework for Accurate Inference with Analog In-Memory Computing](https://mlsanity.com/paper/2305.10459/) (2023)
- [Towards Exact Gradient-based Training on Analog In-memory Computing](https://mlsanity.com/paper/2406.12774/) (2024)
- [Examining the Role and Limits of Batchnorm Optimization to Mitigate Diverse Hardware-noise in In-memory Computing](https://mlsanity.com/paper/2305.18416/) (2023)
- [Mixture of Tokens: Continuous MoE through Cross-Example Aggregation](https://mlsanity.com/paper/2310.15961/) (2023)
- [Analog In-memory Training on General Non-ideal Resistive Elements: The Impact of Response Functions](https://mlsanity.com/paper/2502.06309/) (2025)
- [ClipFormer: Key-Value Clipping of Transformers on Memristive Crossbars for Write Noise Mitigation](https://mlsanity.com/paper/2402.02586/) (2024)
- [Towards training digitally-tied analog blocks via hybrid gradient computation](https://mlsanity.com/paper/2409.03306/) (2024)
- [Measurement-driven neural-network training for integrated magnetic tunnel junction arrays](https://mlsanity.com/paper/2312.06446/) (2023)
- [DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Models](https://mlsanity.com/paper/2404.08079/) (2024)
- [Analog Foundation Models](https://mlsanity.com/paper/2505.09663/) (2025)
- [TReX- Reusing Vision Transformer's Attention for Efficient Xbar-based Computing](https://mlsanity.com/paper/2408.12742/) (2024)
- [Efficient transformer adaptation for analog in-memory computing via low-rank adapters](https://mlsanity.com/paper/2411.17367/) (2024)
- [When In-memory Computing Meets Spiking Neural Networks -- A Perspective on Device-Circuit-System-and-Algorithm Co-design](https://mlsanity.com/paper/2408.12767/) (2024)
- [Harnessing Nonidealities in Analog In-Memory Computing Circuits: A Physical Modeling Approach for Neuromorphic Systems](https://mlsanity.com/paper/2412.09010/) (2024)
- [Onboard Optimization and Learning: A Survey](https://mlsanity.com/paper/2505.08793/) (2025)
- [Rapid yet accurate Tile-circuit and device modeling for Analog In-Memory Computing](https://mlsanity.com/paper/2506.00004/) (2025)
- [AnalogNAS-Bench: A NAS Benchmark for Analog In-Memory Computing](https://mlsanity.com/paper/2506.18495/) (2025)
- [Memristor-Based Neural Network Accelerators for Space Applications: Enhancing Performance with Temporal Averaging and SIRENs](https://mlsanity.com/paper/2509.04506/) (2025)
- [Making Analog Training Scale: Co-Designing Mapping, Optimizer, and Converters](https://mlsanity.com/paper/2609.36584/) (2026)
- [Hyperspectral In-Memory Computing with Optical Frequency Combs and Programmable Optical Memories](https://mlsanity.com/paper/2310.11014/) (2023)
- [U-SWIM: Universal Selective Write-Verify for Computing-in-Memory Neural Accelerators](https://mlsanity.com/paper/2401.05357/) (2023)
- [Learning-to-learn enables rapid learning with phase-change memory-based in-memory computing](https://mlsanity.com/paper/2405.05141/) (2024)
- [Training a multilayer dynamical spintronic network with standard machine learning tools to perform time series classification](https://mlsanity.com/paper/2408.02835/) (2024)
- [Retro-li: Small-Scale Retrieval Augmented Generation Supporting Noisy Similarity Searches and Domain Shift Generalization](https://mlsanity.com/paper/2410.00004/) (2024)
- [Kernel Approximation using Analog In-Memory Computing](https://mlsanity.com/paper/2411.03375/) (2024)
- [Scaling of hardware-compatible perturbative training algorithms](https://mlsanity.com/paper/2501.15403/) (2025)

## Abstract (arXiv)

Analog in-memory computing (AIMC) -- a promising approach for energy-efficient acceleration of deep learning workloads -- computes matrix-vector multiplications (MVMs) but only approximately, due to nonidealities that often are non-deterministic or nonlinear. This can adversely impact the achievable deep neural network (DNN) inference accuracy as compared to a conventional floating point (FP) implementation. While retraining has previously been suggested to improve robustness, prior work has explored only a few DNN topologies, using disparate and overly simplified AIMC hardware models. Here, we use hardware-aware (HWA) training to systematically examine the accuracy of AIMC for multiple common artificial intelligence (AI) workloads across multiple DNN topologies, and investigate sensitivity and robustness to a broad set of nonidealities. By introducing a new and highly realistic AIMC crossbar-model, we improve significantly on earlier retraining approaches. We show that many large-scale DNNs of various topologies, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformers, can in fact be successfully retrained to show iso-accuracy on AIMC. Our results further suggest that AIMC nonidealities that add noise to the inputs or outputs, not the weights, have the largest impact on DNN accuracy, and that RNNs are particularly robust to all nonidealities.