mlsanityData to 7 Oct 2026

NEON: Enabling Efficient Support for Nonlinear Operations in Resistive RAM-based Neural Network Accelerators

arXiv 2211.05730 ↗First posted 10 Nov 2022cs.AR cs.AI cs.ETcited by 0

Not on the map. Why?

Abstract (arXiv)

Resistive Random-Access Memory (RRAM) is well-suited to accelerate neural network (NN) workloads as RRAM-based Processing-in-Memory (PIM) architectures natively support highly-parallel multiply-accumulate (MAC) operations that form the backbone of most NN workloads. Unfortunately, NN workloads such as transformers require support for non-MAC operations (e.g., softmax) that RRAM cannot provide natively. Consequently, state-of-the-art works either integrate additional digital logic circuits to support the non-MAC operations or offload the non-MAC operations to CPU/GPU, resulting in significant performance and energy efficiency overheads due to data movement. In this work, we propose NEON, a novel compiler optimization to enable the end-to-end execution of the NN workload in RRAM. The key idea of NEON is to transform each non-MAC operation into a lightweight yet highly-accurate neural network. Utilizing neural networks to approximate the non-MAC operations provides two advantages: 1) We can exploit the key strength of RRAM, i.e., highly-parallel MAC operation, to flexibly and efficiently execute non-MAC operations in memory. 2) We can simplify RRAM's microarchitecture by eliminating the additional digital logic circuits while reducing the data movement overheads. Acceleration of the non-MAC operations in memory enables NEON to achieve a 2.28x speedup compared to an idealized digital logic-based RRAM. We analyze the trade-offs associated with the transformation and demonstrate feasible use cases for NEON across different substrates.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 48 of its 184 references that are in the map. First posted on arXiv 10 Nov 2022.

Builds on

48 of 184 references.

References · 48

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Layer Normalizationcited by 5,579
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  9. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  10. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  11. 2016End to End Learning for Self-Driving Carscited by 1,000
  12. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
36 more
  1. 2017Dynamic Routing Between Capsulescited by 689
  2. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  3. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  4. 2017Progressive Neural Architecture Searchcited by 483
  5. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  6. 2015Importance Weighted Autoencoderscited by 454
  7. 2017Large-Scale Evolution of Image Classifierscited by 348
  8. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  9. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
  10. 2014cuDNN: Efficient Primitives for Deep Learningcited by 240
  11. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
  12. 2018Auto-Keras: An Efficient Neural Architecture Search Systemcited by 169
  13. 2020FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensionscited by 152
  14. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
  15. 2017Genetic CNNcited by 124
  16. 2018Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attackcited by 123
  17. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  18. 2019AutoGAN: Neural Architecture Search for Generative Adversarial Networkscited by 94
  19. 2019PUMA: A Programmable Ultra-efficient Memristor-based Accelerator for Machine Learning Inferencecited by 57
  20. 2017Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimizationcited by 56
  21. 2019Accuracy vs. Efficiency: Achieving Both through FPGA-Implementation Aware Neural Architecture Searchcited by 38
  22. 2019XNAS: Neural Architecture Search with Expert Advicecited by 35
  23. 2016Distribution-Specific Hardness of Learning Neural Networkscited by 27
  24. 2019EDEN: Enabling Energy-Efficient, High-Performance Deep Neural Network Inference Using Approximate DRAMcited by 24
  25. 2020Neural Architecture Search for Lightweight Non-Local Networkscited by 23
  26. 2021Google Neural Network Models for Edge Devices: Analyzing and Mitigating Machine Learning Inference Bottleneckscited by 22
  27. 2019Processing Data Where It Makes Sense: Enabling In-Memory Computationcited by 21
  28. 2017GraphR: Accelerating Graph Processing Using ReRAMcited by 19
  29. 2019PANTHER: A Programmable Architecture for Neural Network Training Harnessing Energy-efficient ReRAMcited by 16
  30. 2017Multiscale Co-Design Analysis of Energy, Latency, Area, and Accuracy of a ReRAM Analog Neural Training Acceleratorcited by 15
  31. 2019Beyond the Memory Wall: A Case for Memory-centric HPC System for Deep Learningcited by 9
  32. 2022Examining the Robustness of Spiking Neural Networks on Non-ideal Memristive Crossbarscited by 9
  33. 2018CapsAcc: An Efficient Hardware Accelerator for CapsuleNets with Data Reusecited by 4
  34. 2019FPSA: A Full System Stack Solution for Reconfigurable ReRAM-based NN Accelerator Architecturecited by 4
  35. 2019RED: A ReRAM-based Deconvolution Acceleratorcited by 2
  36. 2017Bridging the Gap Between Neural Networks and Neuromorphic Hardware with A Neural Network Compilercited by 1

Built on by

Not cited yet.