# A Survey on Design Methodologies for Accelerating Deep Learning on Heterogeneous Architectures

arXiv: https://arxiv.org/abs/2311.17815 · HTML: https://mlsanity.com/paper/2311.17815/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 216 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Builds on (most cited first, 30 of 36)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [DAPPLE: A Pipelined Data Parallel Approach for Training Large Models](https://mlsanity.com/paper/2007.01045/) (2020)
- [Device Placement Optimization with Reinforcement Learning](https://mlsanity.com/paper/1706.04972/) (2017)
- [Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions](https://mlsanity.com/paper/1802.04730/) (2018)
- [Machine Learning for Electronic Design Automation: A Survey](https://mlsanity.com/paper/2102.03357/) (2021)
- [A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks](https://mlsanity.com/paper/2306.07303/) (2023)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integration](https://mlsanity.com/paper/1911.09925/) (2019)
- [PUMA: A Programmable Ultra-efficient Memristor-based Accelerator for Machine Learning Inference](https://mlsanity.com/paper/1901.10351/) (2019)
- [DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUs](https://mlsanity.com/paper/2008.07127/) (2020)
- [Hardware/Software Co-Exploration of Neural Architectures](https://mlsanity.com/paper/1907.04650/) (2019)
- [Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python](https://mlsanity.com/paper/2112.08429/) (2021)
- [Interstellar: Using Halide's Scheduling Language to Analyze DNN Accelerators](https://mlsanity.com/paper/1809.04070/) (2018)
- [Best of Both Worlds: AutoML Codesign of a CNN and its Hardware Accelerator](https://mlsanity.com/paper/2002.05022/) (2020)
- [Relay: A New IR for Machine Learning Frameworks](https://mlsanity.com/paper/1810.00952/) (2018)
- [Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks](https://mlsanity.com/paper/1802.04924/) (2018)
- [ScaleHLS: A New Scalable High-Level Synthesis Framework on Multi-Level Intermediate Representation](https://mlsanity.com/paper/2107.11673/) (2021)
- [A Survey on Deep Learning Hardware Accelerators for Heterogeneous HPC Platforms](https://mlsanity.com/paper/2306.15552/) (2023)
- [Compiling ONNX Neural Network Models Using MLIR](https://mlsanity.com/paper/2008.08272/) (2020)
- [Sparseloop: An Analytical Approach To Sparse Tensor Accelerator Modeling](https://mlsanity.com/paper/2205.05826/) (2022)
- [DeFiNES: Enabling Fast Exploration of the Depth-first Scheduling Space for DNN Accelerators through Analytical Modeling](https://mlsanity.com/paper/2212.05344/) (2022)
- [Toward Sustainable HPC: Carbon Footprint Estimation and Environmental Implications of HPC Systems](https://mlsanity.com/paper/2306.13177/) (2023)
- [Relax: Composable Abstractions for End-to-End Dynamic Machine Learning](https://mlsanity.com/paper/2311.02103/) (2023)
- [The Effects of Approximate Multiplication on Convolutional Neural Networks](https://mlsanity.com/paper/2007.10500/) (2020)
- [HIDA: A Hierarchical Dataflow Compiler for High-Level Synthesis](https://mlsanity.com/paper/2311.03379/) (2023)

## Abstract (arXiv)

Given their increasing size and complexity, the need for efficient execution of deep neural networks has become increasingly pressing in the design of heterogeneous High-Performance Computing (HPC) and edge platforms, leading to a wide variety of proposals for specialized deep learning architectures and hardware accelerators. The design of such architectures and accelerators requires a multidisciplinary approach combining expertise from several areas, from machine learning to computer architecture, low-level hardware design, and approximate computing. Several methodologies and tools have been proposed to improve the process of designing accelerators for deep learning, aimed at maximizing parallelism and minimizing data movement to achieve high performance and energy efficiency. This paper critically reviews influential tools and design methodologies for Deep Learning accelerators, offering a wide perspective in this rapidly evolving field. This work complements surveys on architectures and accelerators by covering hardware-software co-design, automated synthesis, domain-specific compilers, design space exploration, modeling, and simulation, providing insights into technical challenges and open research directions.