# A Survey on Deep Learning Hardware Accelerators for Heterogeneous HPC Platforms

arXiv: https://arxiv.org/abs/2306.15552 · HTML: https://mlsanity.com/paper/2306.15552/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 226 references that are in the map.
- First posted on arXiv 27 Jun 2023.

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Quantum Machine Learning](https://mlsanity.com/paper/1611.09347/) (2016)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://mlsanity.com/paper/2304.01433/) (2023)
- [Binary Neural Networks: A Survey](https://mlsanity.com/paper/2004.03333/) (2020)
- [FINN: A Framework for Fast, Scalable Binarized Neural Network Inference](https://mlsanity.com/paper/1612.07119/) (2016)
- [SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks](https://mlsanity.com/paper/1708.04485/) (2017)
- [MLPerf Training Benchmark](https://mlsanity.com/paper/1910.01500/) (2019)
- [Full Stack Optimization of Transformer Inference: a Survey](https://mlsanity.com/paper/2302.14017/) (2023)
- [Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation](https://mlsanity.com/paper/2111.14826/) (2021)
- [TensorFlow Quantum: A Software Framework for Quantum Machine Learning](https://mlsanity.com/paper/2003.02989/) (2020)
- [Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integration](https://mlsanity.com/paper/1911.09925/) (2019)
- [Dissecting the Graphcore IPU Architecture via Microbenchmarking](https://mlsanity.com/paper/1912.03413/) (2019)
- [NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps](https://mlsanity.com/paper/1706.01406/) (2017)
- [Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsity](https://mlsanity.com/paper/2008.13006/) (2020)
- [F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization](https://mlsanity.com/paper/2202.05239/) (2022)
- [AI and ML Accelerator Survey and Trends](https://mlsanity.com/paper/2210.04055/) (2022)
- [UCNN: Exploiting Computational Reuse in Deep Neural Networks via Weight Repetition](https://mlsanity.com/paper/1804.06508/) (2018)
- [Vega: A 10-Core SoC for IoT End-Nodes with DNN Acceleration and Cognitive Wake-Up From MRAM-Based State-Retentive Sleep Mode](https://mlsanity.com/paper/2110.09101/) (2021)
- [Flexagon: A Multi-Dataflow Sparse-Sparse Matrix Multiplication Accelerator for Efficient DNN Processing](https://mlsanity.com/paper/2301.10852/) (2023)
- [MorphIC: A 65-nm 738k-Synapse/mm2 Quad-Core Binary-Weight Digital Neuromorphic Processor with Stochastic Spike-Driven Online Learning](https://mlsanity.com/paper/1904.08513/) (2019)
- [PERMDNN: Efficient Compressed DNN Architecture with Permuted Diagonal Matrices](https://mlsanity.com/paper/2004.10936/) (2020)
- [GANAX: A Unified MIMD-SIMD Acceleration for Generative Adversarial Networks](https://mlsanity.com/paper/1806.01107/) (2018)
- [A Systematic Survey of General Sparse Matrix-Matrix Multiplication](https://mlsanity.com/paper/2002.11273/) (2020)

## Built on by (most cited first, 21 of 21)

- [Generative AI for the Optimization of Next-Generation Wireless Networks: Basics, State-of-the-Art, and Open Challenges](https://mlsanity.com/paper/2405.17454/) (2024)
- [LILogic Net: Compact Logic Gate Networks with Learnable Connectivity for Efficient Hardware Deployment](https://mlsanity.com/paper/2511.12340/) (2025)
- [Performance Analysis of Edge and In-Sensor AI Processors: A Comparative Review](https://mlsanity.com/paper/2603.08725/) (2026)
- [Trusting What You Cannot See: Auditable Fine-Tuning and Inference for Proprietary AI](https://mlsanity.com/paper/2603.07466/) (2026)
- [Towards Cold-Start Drafting and Continual Refining: A Value-Driven Memory Approach with Application to NPU Kernel Synthesis](https://mlsanity.com/paper/2603.10846/) (2026)
- [Toward Energy-Efficient Massive MIMO: Graph Neural Network Precoding for Mitigating Non-Linear PA Distortion](https://mlsanity.com/paper/2312.04591/) (2023)
- [An Open-Source HW-SW Co-Development Framework Enabling Efficient Multi-Accelerator Systems](https://mlsanity.com/paper/2508.14582/) (2025)
- [Deep Neural Networks Inspired by Differential Equations](https://mlsanity.com/paper/2510.09685/) (2025)
- [Mathematical Algorithm Design for Deep Learning under Societal and Judicial Constraints: The Algorithmic Transparency Requirement](https://mlsanity.com/paper/2401.10310/) (2024)
- [Performance Measurements in the AI-Centric Computing Continuum Systems](https://mlsanity.com/paper/2506.22884/) (2025)
- [On the accuracy of implicit neural representations for cardiovascular anatomies and hemodynamic fields](https://mlsanity.com/paper/2510.20970/) (2025)
- [A Survey on Design Methodologies for Accelerating Deep Learning on Heterogeneous Architectures](https://mlsanity.com/paper/2311.17815/) (2023)
- [Puzzle: Scheduling Multiple Deep Learning Models on Mobile Device with Heterogeneous Processors](https://mlsanity.com/paper/2508.17764/) (2025)
- [CURA: Size Isnt All You Need -- A Compact Universal Architecture for On-Device Intelligence](https://mlsanity.com/paper/2509.24601/) (2025)
- [Mozart: A Chiplet Ecosystem-Accelerator Codesign Framework for Composable Bespoke Application Specific Integrated Circuits](https://mlsanity.com/paper/2510.08873/) (2025)
- [ScaleDL: Towards Scalable and Efficient Runtime Prediction for Distributed Deep Learning Workloads](https://mlsanity.com/paper/2511.04162/) (2025)
- [Neural surrogates for designing gravitational wave detectors](https://mlsanity.com/paper/2511.19364/) (2025)
- ['1'-bit Count-based Sorting Unit to Reduce Link Power in DNN Accelerators](https://mlsanity.com/paper/2601.14087/) (2026)
- [ALADIN: Accuracy-Latency-Aware Design-space Inference Analysis for Embedded AI Accelerators](https://mlsanity.com/paper/2603.08722/) (2026)
- [AI-RAN on NPUs: Baseband Processing Without Baseband Chips](https://mlsanity.com/paper/2607.04224/) (2026)
- [From Experience to Expertise: Adoption-Aware Memory Learning for Data-Scarce NPU Kernel Synthesis](https://mlsanity.com/paper/2609.35568/) (2026)

## Abstract (arXiv)

Recent trends in deep learning (DL) have made hardware accelerators essential for various high-performance computing (HPC) applications, including image classification, computer vision, and speech recognition. This survey summarizes and classifies the most recent developments in DL accelerators, focusing on their role in meeting the performance demands of HPC applications. We explore cutting-edge approaches to DL acceleration, covering not only GPU- and TPU-based platforms but also specialized hardware such as FPGA- and ASIC-based accelerators, Neural Processing Units, open hardware RISC-V-based accelerators, and co-processors. This survey also describes accelerators leveraging emerging memory technologies and computing paradigms, including 3D-stacked Processor-In-Memory, non-volatile memories like Resistive RAM and Phase Change Memories used for in-memory computing, as well as Neuromorphic Processing Units, and Multi-Chip Module-based accelerators. Furthermore, we provide insights into emerging quantum-based accelerators and photonics. Finally, this survey categorizes the most influential architectures and technologies from recent years, offering readers a comprehensive perspective on the rapidly evolving field of deep learning acceleration.