# DNNVM : End-to-End Compiler Leveraging Heterogeneous Optimizations on FPGA-based CNN Accelerators

arXiv: https://arxiv.org/abs/1902.07463 · HTML: https://mlsanity.com/paper/1902.07463/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2019.

## Built on by (most cited first, 6 of 6)

- [MATCH: Model-Aware TVM-based Compilation for Heterogeneous Edge Devices](https://mlsanity.com/paper/2410.08855/) (2024)
- [unzipFPGA: Enhancing FPGA-based CNN Engines with On-the-Fly Weights Generation](https://mlsanity.com/paper/2103.05600/) (2021)
- [eIQ Neutron: Redefining Edge-AI Inference with Integrated NPU and Compiler Innovations](https://mlsanity.com/paper/2509.14388/) (2025)
- [How to Reach Real-Time AI on Consumer Devices? Solutions for Programmable and Custom Architectures](https://mlsanity.com/paper/2106.15021/) (2021)
- [Mitigating Memory Wall Effects in CNN Engines with On-the-Fly Weights Generation](https://mlsanity.com/paper/2307.13412/) (2023)
- [A Real-time Endoscopic Image Denoising System](https://mlsanity.com/paper/2506.15395/) (2025)

## Abstract (arXiv)

The convolutional neural network (CNN) has become a state-of-the-art method for several artificial intelligence domains in recent years. The increasingly complex CNN models are both computation-bound and I/O-bound. FPGA-based accelerators driven by custom instruction set architecture (ISA) achieve a balance between generality and efficiency, but there is much on them left to be optimized. We propose the full-stack compiler DNNVM, which is an integration of optimizers for graphs, loops and data layouts, and an assembler, a runtime supporter and a validation environment. The DNNVM works in the context of deep learning frameworks and transforms CNN models into the directed acyclic graph: XGraph. Based on XGraph, we transform the optimization challenges for both the data layout and pipeline into graph-level problems. DNNVM enumerates all potentially profitable fusion opportunities by a heuristic subgraph isomorphism algorithm to leverage pipeline and data layout optimizations, and searches for the best choice of execution strategies of the whole computing graph. On the Xilinx ZU2 @330 MHz and ZU9 @330 MHz, we achieve equivalently state-of-the-art performance on our benchmarks by na"ive implementations without optimizations, and the throughput is further improved up to 1.26x by leveraging heterogeneous optimizations in DNNVM. Finally, with ZU9 @330 MHz, we achieve state-of-the-art performance for VGG and ResNet50. We achieve a throughput of 2.82 TOPs/s and an energy efficiency of 123.7 GOPs/s/W for VGG. Additionally, we achieve 1.38 TOPs/s for ResNet50 and 1.41 TOPs/s for GoogleNet.