# Achieving Super-Linear Speedup across Multi-FPGA for Real-Time DNN Inference

arXiv: https://arxiv.org/abs/1907.08985 · HTML: https://mlsanity.com/paper/1907.08985/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2019.

## Built on by (most cited first, 13 of 13)

- [Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimization](https://mlsanity.com/paper/2110.10030/) (2021)
- [Auditing Membership Leakages of Multi-Exit Networks](https://mlsanity.com/paper/2208.11180/) (2022)
- [The Larger The Fairer? Small Neural Networks Can Achieve Fairness for Edge Devices](https://mlsanity.com/paper/2202.11317/) (2022)
- [Can Noise on Qubits Be Learned in Quantum Neural Network? A Case Study on QuantumFlow](https://mlsanity.com/paper/2109.03430/) (2021)
- [Exploration of Quantum Neural Architecture by Mixing Quantum Neuron Designs](https://mlsanity.com/paper/2109.03806/) (2021)
- [On-Device Unsupervised Image Segmentation](https://mlsanity.com/paper/2303.12753/) (2023)
- [Dancing along Battery: Enabling Transformer with Run-time Reconfigurability on Mobile Devices](https://mlsanity.com/paper/2102.06336/) (2021)
- [EF-Train: Enable Efficient On-device CNN Training on FPGA Through Data Reshaping for Online Adaptation or Personalization](https://mlsanity.com/paper/2202.10935/) (2022)
- [H2H: Heterogeneous Model to Heterogeneous System Mapping with Computation and Communication Awareness](https://mlsanity.com/paper/2204.13852/) (2022)
- [Brain-inspired Cognition in Next Generation Racetrack Memories](https://mlsanity.com/paper/2111.02246/) (2021)
- [A Semi-Decoupled Approach to Fast and Optimal Hardware-Software Co-Design of Neural Accelerators](https://mlsanity.com/paper/2203.13921/) (2022)
- [A Model Stealing Attack Against Multi-Exit Networks](https://mlsanity.com/paper/2305.13584/) (2023)
- [MARS: Exploiting Multi-Level Parallelism for DNN Workloads on Adaptive Multi-Accelerator Systems](https://mlsanity.com/paper/2307.12234/) (2023)

## Abstract (arXiv)

Real-time Deep Neural Network (DNN) inference with low-latency requirement has become increasingly important for numerous applications in both cloud computing (e.g., Apple's Siri) and edge computing (e.g., Google/Waymo's driverless car). FPGA-based DNN accelerators have demonstrated both superior flexibility and performance; in addition, for real-time inference with low batch size, FPGA is expected to achieve further performance improvement. However, the performance gain from the single-FPGA design is obstructed by the limited on-chip resource. In this paper, we employ multiple FPGAs to cooperatively run DNNs with the objective of achieving super-linear speed-up against single-FPGA design. In implementing such systems, we found two barriers that hinder us from achieving the design goal: (1) the lack of a clear partition scheme for each DNN layer to fully exploit parallelism, and (2) the insufficient bandwidth between the off-chip memory and the accelerator due to the growing size of DNNs. To tackle these issues, we propose a general framework, "Super-LIP", which can support different kinds of DNNs. In this paper, we take Convolutional Neural Network (CNN) as a vehicle to illustrate Super-LIP. We first formulate an accurate system-level model to support the exploration of best partition schemes. Then, we develop a novel design methodology to effectively alleviate the heavy loads on memory bandwidth by moving traffic from memory bus to inter-FPGA links. We implement Super-LIP based on ZCU102 FPGA boards. Results demonstrate that Super-LIP with 2 FPGAs can achieve 3.48x speedup, compared to the state-of-the-art single-FPGA design. What is more, as the number of FPGAs scales up, the system latency can be further reduced while maintaining high energy efficiency.