mlsanityData to 14 Oct 2026

Fast convolutional neural networks on FPGAs with hls4ml

arXiv 2101.05108 ↗First posted 13 Jan 2021cs.LG cs.CV hep-excited by 42

Not on the map. Why?

Abstract (arXiv)

We introduce an automated tool for deploying ultra low-latency, low-power deep neural networks with convolutional layers on FPGAs. By extending the hls4ml library, we demonstrate an inference latency of 5,μs using convolutional architectures, targeting microsecond latency applications like those at the CERN Large Hadron Collider. Considering benchmark models trained on the Street View House Numbers Dataset, we demonstrate various methods for model compression in order to fit the computational constraints of a typical FPGA device used in trigger and data acquisition systems of particle detectors. In particular, we discuss pruning and quantization-aware training, and demonstrate how resource utilization can be significantly reduced with little to no loss in model accuracy. We show that the FPGA critical resource consumption can be reduced by 97% with zero loss in model accuracy, and by 99% when tolerating a 6% accuracy degradation.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 36 of its 70 references that are in the map. First posted on arXiv 13 Jan 2021.

Builds on

36 of 70 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2016Wide Residual Networkscited by 3,070
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  8. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  10. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  11. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  12. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
24 more
  1. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  2. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  3. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 288
  4. 2016FINN: A Framework for Fast, Scalable Binarized Neural Network Inferencecited by 150
  5. 2016Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruningcited by 134
  6. 2018Fast inference of deep neural networks in FPGAs for particle physicscited by 133
  7. 2020Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectorscited by 80
  8. 2018FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networkscited by 56
  9. 2020Compressing deep neural networks on FPGAs to binary and ternary precision with HLS4MLcited by 42
  10. 2017NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Mapscited by 39
  11. 2017The power of deeper networks for expressing natural functionscited by 35
  12. 2020Distance-Weighted Graph Neural Networks on FPGAs for Real-Time Particle Reconstruction in High Energy Physicscited by 34
  13. 2016Convolutional Rectifier Networks as Generalized Tensor Decompositionscited by 32
  14. 2017Parallel Multi Channel Convolution using General Matrix Multiplicationcited by 30
  15. 2019FPGA-based Accelerators of Deep Learning Networks for Learning and Classification: A Reviewcited by 29
  16. 2018Toolflows for Mapping Convolutional Neural Networks on FPGAs: A Survey and Future Directionscited by 27
  17. 2020Fast inference of Boosted Decision Trees in FPGAs for particle physicscited by 26
  18. 2020Accelerated Charged Particle Tracking with Graph Neural Networks on FPGAscited by 23
  19. 2018Accelerating CNN inference on FPGAs: A Surveycited by 16
  20. 2016Residual Networks of Residual Networks: Multilevel Residual Networkscited by 13
  21. 2016Caffeinated FPGAs: FPGA Framework For Convolutional Neural Networkscited by 8
  22. 2017fpgaConvNet: A Toolflow for Mapping Diverse Convolutional Neural Networks on Embedded FPGAscited by 5
  23. 2017Snowflake: A Model Agnostic Accelerator for Deep Convolutional Neural Networkscited by 3
  24. 2019A flexible FPGA accelerator for convolutional neural networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 8; 2023: 5; 2024: 10; 2025: 8; 2026: 4.

42 papers.

Sort:

  1. 2021hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devicescited by 62
  2. 2021Machine Learning in the Search for New Fundamental Physicscited by 50
  3. 2021Applications and Techniques for Fast Machine Learning in Sciencecited by 32
  4. 2021Charged particle tracking via edge-classifying interaction networkscited by 19
  5. 2021Graph Neural Networks for Charged Particle Tracking on FPGAscited by 17
  6. 2021Online-compatible Unsupervised Non-resonant Anomaly Detectioncited by 15
  7. 2022Ultra-low latency recurrent neural network inference on FPGAs for physics applications with hls4mlcited by 14
  8. 2022Graph Neural Networks in Particle Physics: Implementations, Innovations, and Challengescited by 12
  9. 2022Real-time semantic segmentation on FPGAs for autonomous vehicles with hls4mlcited by 11
  10. 2022Snowmass 2021 Computational Frontier CompF03 Topical Group Report: Machine Learningcited by 10
  11. 2021Nanosecond machine learning event classification with boosted decision trees in FPGA for high energy physicscited by 9
  12. 2022Open-source FPGA-ML codesign for the MLPerf Tiny Benchmarkcited by 9
30 more
  1. 2024HGQ: High Granularity Quantization for Real-time Neural Networks on FPGAscited by 9
  2. 2025hls4ml: A Flexible, Open-Source Platform for Deep Learning Acceleration on Reconfigurable Hardwarecited by 9
  3. 2025da4ml: Distributed Arithmetic for Real-time Neural Networks on FPGAscited by 8
  4. 2023FPGA Resource-aware Structured Pruning for Real-Time Neural Networkscited by 7
  5. 2023Low latency optical-based mode tracking with machine learning deployed on FPGAs on a tokamakcited by 7
  6. 2024SymbolNet: Neural Symbolic Regression with Adaptive Dynamic Pruning for Compressioncited by 7
  7. 2024Real-time Anomaly Detection at the L1 Trigger of CMS Experimentcited by 7
  8. 2023Tailor: Altering Skip Connections for Resource-Efficient Inferencecited by 6
  9. 2024Ultra Fast Transformers on FPGAs for Particle Physics Experimentscited by 6
  10. 2025WARP-LUTs -- Walsh-Assisted Relaxation for Probabilistic Look Up Tablescited by 6
  11. 2023Fast Neural Network Inference on FPGAs for Triggering on Long-Lived Particles at Colliderscited by 5
  12. 2024Low Latency Transformer Inference on FPGAs for Physics Applications with hls4mlcited by 5
  13. 2022Data-Model-Circuit Tri-Design for Ultra-Light Video Intelligence on Edge Devicescited by 4
  14. 2024Ultra-low latency quantum-inspired machine learning predictors implemented on FPGAcited by 4
  15. 2024rule4ml: An Open-Source Tool for Resource Utilization and Latency Estimation for ML Models on FPGAcited by 3
  16. 2026JetFormer: A Scalable and Efficient Transformer for Jet Tagging from Offline Analysis to FPGA Triggerscited by 3
  17. 2022Physics Community Needs, Tools, and Resources for Machine Learningcited by 2
  18. 2022FastStamp: Accelerating Neural Steganography and Digital Watermarking of Images on FPGAscited by 2
  19. 2023End-to-end codesign of Hessian-aware quantized neural networks for FPGAs and ASICscited by 2
  20. 2026WARP Logic Neural Networkscited by 2
  21. 2024Efficient Edge AI: Deploying Convolutional Neural Networks on FPGA with the Gemmini Acceleratorcited by 1
  22. 2025Real-Time Cell Sorting with Scalable In Situ FPGA-Accelerated Deep Learningcited by 1
  23. 2025BitHEP -- The Limits of Low-Precision ML in HEPcited by 1
  24. 2026Spiking Neural Network inference on FPGAs with hls4mlcited by 1
  25. 2024FPGA Deployment of LFADS for Real-time Neuroscience Experimentscited by 0
  26. 2024Scintillation pulse characterization with spectrum-inspired temporal neural networks: case studies on particle detector signalscited by 0
  27. 2025Physics-Informed Graph Neural Networks for Transverse Momentum Estimation in CMS Trigger Systemscited by 0
  28. 2025Neural Network Acceleration on MPSoC board: Integrating SLAC's SNL, Rogue Software and Auto-SNLcited by 0
  29. 2025SparsePixels: Efficient Convolution for Sparse Data on FPGAscited by 0
  30. 2026PQuantML: A Tool for End-to-End Hardware-aware Model Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Enabling Low-Latency Machine learning on Radiation-Hard FPGAs with hls4ml
  2. 2023Design and Optimization of Residual Neural Network Accelerators for Low-Power FPGAs Using High-Level Synthesis ↗
  3. 2025Sub-microsecond Transformers for Jet Tagging on FPGAs
  4. 2026Towards Tensor Network Models for Low-Latency Jet Tagging on FPGAs
  5. 2021A Competitive Edge: Can FPGAs Beat GPUs at DCNN Inference Acceleration in Resource-Limited Edge Computing Applications?
  6. 2025Neural Architecture Codesign for Fast Physics Applications
  7. 2025Energy-Efficient FPGA Framework for Non-Quantized Convolutional Neural Networks
  8. 2024A survey on FPGA-based accelerator for ML models