# Fast convolutional neural networks on FPGAs with hls4ml

arXiv: https://arxiv.org/abs/2101.05108 · HTML: https://mlsanity.com/paper/2101.05108/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 70 references that are in the map.
- First posted on arXiv 13 Jan 2021.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Caffe: Convolutional Architecture for Fast Feature Embedding](https://mlsanity.com/paper/1408.5093/) (2014)
- [FINN: A Framework for Fast, Scalable Binarized Neural Network Inference](https://mlsanity.com/paper/1612.07119/) (2016)
- [Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning](https://mlsanity.com/paper/1611.05128/) (2016)
- [Fast inference of deep neural networks in FPGAs for particle physics](https://mlsanity.com/paper/1804.06913/) (2018)
- [Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors](https://mlsanity.com/paper/2006.10159/) (2020)
- [FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networks](https://mlsanity.com/paper/1809.04570/) (2018)
- [Compressing deep neural networks on FPGAs to binary and ternary precision with HLS4ML](https://mlsanity.com/paper/2003.06308/) (2020)
- [NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps](https://mlsanity.com/paper/1706.01406/) (2017)
- [The power of deeper networks for expressing natural functions](https://mlsanity.com/paper/1705.05502/) (2017)
- [Distance-Weighted Graph Neural Networks on FPGAs for Real-Time Particle Reconstruction in High Energy Physics](https://mlsanity.com/paper/2008.03601/) (2020)
- [Convolutional Rectifier Networks as Generalized Tensor Decompositions](https://mlsanity.com/paper/1603.00162/) (2016)
- [Parallel Multi Channel Convolution using General Matrix Multiplication](https://mlsanity.com/paper/1704.04428/) (2017)
- [FPGA-based Accelerators of Deep Learning Networks for Learning and Classification: A Review](https://mlsanity.com/paper/1901.00121/) (2019)
- [Toolflows for Mapping Convolutional Neural Networks on FPGAs: A Survey and Future Directions](https://mlsanity.com/paper/1803.05900/) (2018)
- [Fast inference of Boosted Decision Trees in FPGAs for particle physics](https://mlsanity.com/paper/2002.02534/) (2020)
- [Accelerated Charged Particle Tracking with Graph Neural Networks on FPGAs](https://mlsanity.com/paper/2012.01563/) (2020)

## Built on by (most cited first, 30 of 42)

- [hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devices](https://mlsanity.com/paper/2103.05579/) (2021)
- [Machine Learning in the Search for New Fundamental Physics](https://mlsanity.com/paper/2112.03769/) (2021)
- [Applications and Techniques for Fast Machine Learning in Science](https://mlsanity.com/paper/2110.13041/) (2021)
- [Charged particle tracking via edge-classifying interaction networks](https://mlsanity.com/paper/2103.16701/) (2021)
- [Graph Neural Networks for Charged Particle Tracking on FPGAs](https://mlsanity.com/paper/2112.02048/) (2021)
- [Online-compatible Unsupervised Non-resonant Anomaly Detection](https://mlsanity.com/paper/2111.06417/) (2021)
- [Ultra-low latency recurrent neural network inference on FPGAs for physics applications with hls4ml](https://mlsanity.com/paper/2207.00559/) (2022)
- [Graph Neural Networks in Particle Physics: Implementations, Innovations, and Challenges](https://mlsanity.com/paper/2203.12852/) (2022)
- [Real-time semantic segmentation on FPGAs for autonomous vehicles with hls4ml](https://mlsanity.com/paper/2205.07690/) (2022)
- [Snowmass 2021 Computational Frontier CompF03 Topical Group Report: Machine Learning](https://mlsanity.com/paper/2209.07559/) (2022)
- [Nanosecond machine learning event classification with boosted decision trees in FPGA for high energy physics](https://mlsanity.com/paper/2104.03408/) (2021)
- [Open-source FPGA-ML codesign for the MLPerf Tiny Benchmark](https://mlsanity.com/paper/2206.11791/) (2022)
- [HGQ: High Granularity Quantization for Real-time Neural Networks on FPGAs](https://mlsanity.com/paper/2405.00645/) (2024)
- [hls4ml: A Flexible, Open-Source Platform for Deep Learning Acceleration on Reconfigurable Hardware](https://mlsanity.com/paper/2512.01463/) (2025)
- [da4ml: Distributed Arithmetic for Real-time Neural Networks on FPGAs](https://mlsanity.com/paper/2507.04535/) (2025)
- [FPGA Resource-aware Structured Pruning for Real-Time Neural Networks](https://mlsanity.com/paper/2308.05170/) (2023)
- [Low latency optical-based mode tracking with machine learning deployed on FPGAs on a tokamak](https://mlsanity.com/paper/2312.00128/) (2023)
- [SymbolNet: Neural Symbolic Regression with Adaptive Dynamic Pruning for Compression](https://mlsanity.com/paper/2401.09949/) (2024)
- [Real-time Anomaly Detection at the L1 Trigger of CMS Experiment](https://mlsanity.com/paper/2411.19506/) (2024)
- [Tailor: Altering Skip Connections for Resource-Efficient Inference](https://mlsanity.com/paper/2301.07247/) (2023)
- [Ultra Fast Transformers on FPGAs for Particle Physics Experiments](https://mlsanity.com/paper/2402.01047/) (2024)
- [WARP-LUTs -- Walsh-Assisted Relaxation for Probabilistic Look Up Tables](https://mlsanity.com/paper/2510.15655/) (2025)
- [Fast Neural Network Inference on FPGAs for Triggering on Long-Lived Particles at Colliders](https://mlsanity.com/paper/2307.05152/) (2023)
- [Low Latency Transformer Inference on FPGAs for Physics Applications with hls4ml](https://mlsanity.com/paper/2409.05207/) (2024)
- [Data-Model-Circuit Tri-Design for Ultra-Light Video Intelligence on Edge Devices](https://mlsanity.com/paper/2210.08578/) (2022)
- [Ultra-low latency quantum-inspired machine learning predictors implemented on FPGA](https://mlsanity.com/paper/2409.16075/) (2024)
- [rule4ml: An Open-Source Tool for Resource Utilization and Latency Estimation for ML Models on FPGA](https://mlsanity.com/paper/2408.05314/) (2024)
- [JetFormer: A Scalable and Efficient Transformer for Jet Tagging from Offline Analysis to FPGA Triggers](https://mlsanity.com/paper/2601.17215/) (2026)
- [Physics Community Needs, Tools, and Resources for Machine Learning](https://mlsanity.com/paper/2203.16255/) (2022)
- [FastStamp: Accelerating Neural Steganography and Digital Watermarking of Images on FPGAs](https://mlsanity.com/paper/2209.12391/) (2022)

## Abstract (arXiv)

We introduce an automated tool for deploying ultra low-latency, low-power deep neural networks with convolutional layers on FPGAs. By extending the hls4ml library, we demonstrate an inference latency of 5,μs using convolutional architectures, targeting microsecond latency applications like those at the CERN Large Hadron Collider. Considering benchmark models trained on the Street View House Numbers Dataset, we demonstrate various methods for model compression in order to fit the computational constraints of a typical FPGA device used in trigger and data acquisition systems of particle detectors. In particular, we discuss pruning and quantization-aware training, and demonstrate how resource utilization can be significantly reduced with little to no loss in model accuracy. We show that the FPGA critical resource consumption can be reduced by 97% with zero loss in model accuracy, and by 99% when tolerating a 6% accuracy degradation.