# NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps

arXiv: https://arxiv.org/abs/1706.01406 · HTML: https://mlsanity.com/paper/1706.01406/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2017.

## Built on by (most cited first, 30 of 39)

- [AEGNN: Asynchronous Event-based Graph Neural Networks](https://mlsanity.com/paper/2203.17149/) (2022)
- [Fast convolutional neural networks on FPGAs with hls4ml](https://mlsanity.com/paper/2101.05108/) (2021)
- [Dual-side Sparse Tensor Core](https://mlsanity.com/paper/2105.09564/) (2021)
- [Exploiting Spatial Sparsity for Event Cameras with Visual Transformers](https://mlsanity.com/paper/2202.05054/) (2022)
- [A Survey on Deep Learning Hardware Accelerators for Heterogeneous HPC Platforms](https://mlsanity.com/paper/2306.15552/) (2023)
- [Spartus: A 9.4 TOp/s FPGA-based LSTM Accelerator Exploiting Spatio-Temporal Sparsity](https://mlsanity.com/paper/2108.02297/) (2021)
- [Event-Based Eye Tracking. AIS 2024 Challenge Survey](https://mlsanity.com/paper/2404.11770/) (2024)
- [ROBIN: A Robust Optical Binary Neural Network Accelerator](https://mlsanity.com/paper/2107.05530/) (2021)
- [Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networks](https://mlsanity.com/paper/2112.13896/) (2021)
- [Flex-PE: Flexible and SIMD Multi-Precision Processing Element for AI Workloads](https://mlsanity.com/paper/2412.11702/) (2024)
- [SONIC: A Sparse Neural Network Inference Accelerator with Silicon Photonics for Energy-Efficient Deep Learning](https://mlsanity.com/paper/2109.04459/) (2021)
- [Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques](https://mlsanity.com/paper/2112.15131/) (2021)
- [Co-designing a Sub-millisecond Latency Event-based Eye Tracking System with Submanifold Sparse CNN](https://mlsanity.com/paper/2404.14279/) (2024)
- [Event-based vision on FPGAs -- a survey](https://mlsanity.com/paper/2407.08356/) (2024)
- [Event Camera Meets Mobile Embodied Perception: Abstraction, Algorithm, Acceleration, Application](https://mlsanity.com/paper/2503.22943/) (2025)
- [ThreshNet: An Efficient DenseNet Using Threshold Mechanism to Reduce Connections](https://mlsanity.com/paper/2201.03013/) (2022)
- [GNNIE: GNN Inference Engine with Load-balancing and Graph-Specific Caching](https://mlsanity.com/paper/2105.10554/) (2021)
- [Cross-Layer Design for AI Acceleration with Non-Coherent Optical Computing](https://mlsanity.com/paper/2303.12910/) (2023)
- [HYDRA: Hybrid Data Multiplexing and Run-time Layer Configurable DNN Accelerator](https://mlsanity.com/paper/2409.04976/) (2024)
- [Training for temporal sparsity in deep neural networks, application in video processing](https://mlsanity.com/paper/2107.07305/) (2021)
- [An Ultra-low Power TinyML System for Real-time Visual Processing at Edge](https://mlsanity.com/paper/2207.04663/) (2022)
- [Improved Projection Learning for Lower Dimensional Feature Maps](https://mlsanity.com/paper/2210.15170/) (2022)
- [Synapse Compression for Event-Based Convolutional-Neural-Network Accelerators](https://mlsanity.com/paper/2112.07019/) (2021)
- [EcoFlow: Efficient Convolutional Dataflows for Low-Power Neural Network Accelerators](https://mlsanity.com/paper/2202.02310/) (2022)
- [Algorithm and Hardware Co-Design of Energy-Efficient LSTM Networks for Video Recognition with Hierarchical Tucker Tensor Decomposition](https://mlsanity.com/paper/2212.02046/) (2022)
- [Invariance Measures for Neural Networks](https://mlsanity.com/paper/2310.17404/) (2023)
- [A Construction Kit for Efficient Low Power Neural Network Accelerator Designs](https://mlsanity.com/paper/2106.12810/) (2021)
- [Multiply-and-Fire (MNF): An Event-driven Sparse Neural Network Accelerator](https://mlsanity.com/paper/2204.09797/) (2022)
- [Efficient Adaptive Federated Optimization of Federated Learning for IoT](https://mlsanity.com/paper/2206.11448/) (2022)
- [SBPF: Sensitiveness Based Pruning Framework For Convolutional Neural Network On Image Classification](https://mlsanity.com/paper/2208.04588/) (2022)

## Abstract (arXiv)

Convolutional neural networks (CNNs) have become the dominant neural network architecture for solving many state-of-the-art (SOA) visual processing tasks. Even though Graphical Processing Units (GPUs) are most often used in training and deploying CNNs, their power efficiency is less than 10 GOp/s/W for single-frame runtime inference. We propose a flexible and efficient CNN accelerator architecture called NullHop that implements SOA CNNs useful for low-power and low-latency application scenarios. NullHop exploits the sparsity of neuron activations in CNNs to accelerate the computation and reduce memory requirements. The flexible architecture allows high utilization of available computing resources across kernel sizes ranging from 1x1 to 7x7. NullHop can process up to 128 input and 128 output feature maps per layer in a single pass. We implemented the proposed architecture on a Xilinx Zynq FPGA platform and present results showing how our implementation reduces external memory transfers and compute time in five different CNNs ranging from small ones up to the widely known large VGG16 and VGG19 CNNs. Post-synthesis simulations using Mentor Modelsim in a 28nm process with a clock frequency of 500 MHz show that the VGG19 network achieves over 450 GOp/s. By exploiting sparsity, NullHop achieves an efficiency of 368%, maintains over 98% utilization of the MAC units, and achieves a power efficiency of over 3TOp/s/W in a core area of 6.3mm2. As further proof of NullHop's usability, we interfaced its FPGA implementation with a neuromorphic event camera for real time interactive demonstrations.