# GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent

arXiv: https://arxiv.org/abs/2102.07511 · HTML: https://mlsanity.com/paper/2102.07511/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 108 references that are in the map.
- First posted on arXiv 15 Feb 2021.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Horovod: fast and easy distributed deep learning in TensorFlow](https://mlsanity.com/paper/1802.05799/) (2018)
- [cuDNN: Efficient Primitives for Deep Learning](https://mlsanity.com/paper/1410.0759/) (2014)
- [Training Deep Neural Networks with 8-bit Floating Point Numbers](https://mlsanity.com/paper/1812.08011/) (2018)
- [SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks](https://mlsanity.com/paper/1708.04485/) (2017)
- [Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes](https://mlsanity.com/paper/1807.11205/) (2018)
- [PUMA: A Programmable Ultra-efficient Memristor-based Accelerator for Machine Learning Inference](https://mlsanity.com/paper/1901.10351/) (2019)
- [Mixed Precision Training of Convolutional Neural Networks using Integer Operations](https://mlsanity.com/paper/1802.00930/) (2018)
- [TensorDIMM: A Practical Near-Memory Processing Architecture for Embeddings and Tensor Operations in Deep Learning](https://mlsanity.com/paper/1908.03072/) (2019)
- [Mixed Precision Training With 8-bit Floating Point](https://mlsanity.com/paper/1905.12334/) (2019)
- [Restructuring Batch Normalization to Accelerate CNN Training](https://mlsanity.com/paper/1807.01702/) (2018)
- [Distributed Deep Learning Using Synchronous Stochastic Gradient Descent](https://mlsanity.com/paper/1602.06709/) (2016)
- [Mini-batch Serialization: CNN Training with Inter-layer Data Reuse](https://mlsanity.com/paper/1810.00307/) (2018)
- [Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash](https://mlsanity.com/paper/1811.05233/) (2018)

## Built on by (most cited first, 10 of 10)

- [Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System](https://mlsanity.com/paper/2403.06664/) (2024)
- [Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputation](https://mlsanity.com/paper/2209.00606/) (2022)
- [Design Principles for Lifelong Learning AI Accelerators](https://mlsanity.com/paper/2310.04467/) (2023)
- [Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving](https://mlsanity.com/paper/2507.10178/) (2025)
- [PIM-Opt: Demystifying Distributed Optimization Algorithms on a Real-World Processing-In-Memory System](https://mlsanity.com/paper/2404.07164/) (2024)
- [CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIM](https://mlsanity.com/paper/2504.17584/) (2025)
- [A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs](https://mlsanity.com/paper/2502.09921/) (2025)
- [Demystifying BERT: Implications for Accelerator Design](https://mlsanity.com/paper/2104.08335/) (2021)
- [An Experimental Evaluation of Machine Learning Training on a Real Processing-in-Memory System](https://mlsanity.com/paper/2207.07886/) (2022)
- [LOCALUT: Harnessing Capacity-Computation Tradeoffs for LUT-Based Inference in DRAM-PIM](https://mlsanity.com/paper/2604.04523/) (2026)

## Abstract (arXiv)

In this paper, we present GradPIM, a processing-in-memory architecture which accelerates parameter updates of deep neural networks training. As one of processing-in-memory techniques that could be realized in the near future, we propose an incremental, simple architectural design that does not invade the existing memory protocol. Extending DDR4 SDRAM to utilize bank-group parallelism makes our operation designs in processing-in-memory (PIM) module efficient in terms of hardware cost and performance. Our experimental results show that the proposed architecture can improve the performance of DNN training and greatly reduce memory bandwidth requirement while posing only a minimal amount of overhead to the protocol and DRAM area.