# H2PIPE: High throughput CNN Inference on FPGAs with High-Bandwidth Memory

arXiv: https://arxiv.org/abs/2408.09209 · HTML: https://mlsanity.com/paper/2408.09209/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 34 references that are in the map.
- First posted on arXiv 17 Aug 2024.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation](https://mlsanity.com/paper/2004.09602/) (2020)
- [FINN: A Framework for Fast, Scalable Binarized Neural Network Inference](https://mlsanity.com/paper/1612.07119/) (2016)
- [Accelerating CNN inference on FPGAs: A Survey](https://mlsanity.com/paper/1806.01683/) (2018)
- [DLA: Compiler and FPGA Overlay for Neural Network Inference Acceleration](https://mlsanity.com/paper/1807.06434/) (2018)
- [Memory-Efficient Dataflow Inference for Deep CNNs on FPGA](https://mlsanity.com/paper/2011.07317/) (2020)
- [SATAY: A Streaming Architecture Toolflow for Accelerating YOLO Models on FPGA Devices](https://mlsanity.com/paper/2309.01587/) (2023)
- [AoCStream: All-on-Chip CNN Accelerator With Stream-Based Line-Buffer Architecture](https://mlsanity.com/paper/2212.11438/) (2022)
- [Mitigating Memory Wall Effects in CNN Engines with On-the-Fly Weights Generation](https://mlsanity.com/paper/2307.13412/) (2023)

## Built on by (most cited first, 3 of 3)

- [DPUV4E: High-Throughput DPU Architecture Design for CNN on Versal ACAP](https://mlsanity.com/paper/2506.11441/) (2025)
- [Automating Hardware Design and Verification from Architectural Papers via a Neural-Symbolic Graph Framework](https://mlsanity.com/paper/2511.06067/) (2025)
- [Sparse by Command: Task-Conditional Compute Skipping for Multi-Task Inference Accelerators](https://mlsanity.com/paper/2607.22038/) (2026)

## Abstract (arXiv)

Convolutional Neural Networks (CNNs) combine large amounts of parallelizable computation with frequent memory access. Field Programmable Gate Arrays (FPGAs) can achieve low latency and high throughput CNN inference by implementing dataflow accelerators that pipeline layer-specific hardware to implement an entire network. By implementing a different processing element for each CNN layer, these layer-pipelined accelerators can achieve high compute density, but having all layers processing in parallel requires high memory bandwidth. Traditionally this has been satisfied by storing all weights on chip, but this is infeasible for the largest CNNs, which are often those most in need of acceleration. In this work we augment a state-of-the-art dataflow accelerator (HPIPE) to leverage both High-Bandwidth Memory (HBM) and on-chip storage, enabling high performance layer-pipelined dataflow acceleration of large CNNs. Based on profiling results of HBM's latency and throughput against expected address patterns, we develop an algorithm to choose which weight buffers should be moved off chip and how deep the on-chip FIFOs to HBM should be to minimize compute unit stalling. We integrate the new hardware generation within the HPIPE domain-specific CNN compiler and demonstrate good bandwidth efficiency against theoretical limits. Compared to the best prior work we obtain speed-ups of at least 19.4x, 5.1x and 10.5x on ResNet-18, ResNet-50 and VGG-16 respectively.