mlsanityData to 7 Oct 2026

Real-time semantic segmentation on FPGAs for autonomous vehicles with hls4ml

arXiv 2205.07690 ↗First posted 16 May 2022cs.CV cs.AR cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate how field programmable gate arrays can serve as hardware accelerators for real-time semantic segmentation tasks relevant for autonomous driving. Considering compressed versions of the ENet convolutional neural network architecture, we demonstrate a fully-on-chip deployment with a latency of 4.9 ms per image, using less than 30% of the available resources on a Xilinx ZCU102 evaluation board. The latency is reduced to 3 ms per image when increasing the batch size to ten, corresponding to the use case where the autonomous vehicle receives inputs from multiple cameras simultaneously. We show, through aggressive filter reduction and heterogeneous quantization-aware training, and an optimized implementation of convolutional layers, that the power consumption and resource utilization can be significantly reduced while maintaining accuracy on the Cityscapes dataset.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 13 of its 25 references that are in the map. First posted on arXiv 16 May 2022.

Builds on

13 of 25 references.

References · 13

  1. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  2. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  3. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  4. 2016ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentationcited by 291
  5. 2018Fast inference of deep neural networks in FPGAs for particle physicscited by 132
  6. 2020Benchmarking TinyML Systems: Challenges and Directioncited by 89
  7. 2020Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectorscited by 78
  8. 2021hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devicescited by 61
  9. 2020Compressing deep neural networks on FPGAs to binary and ternary precision with HLS4MLcited by 42
  10. 2021Fast convolutional neural networks on FPGAs with hls4mlcited by 42
  11. 2020Distance-Weighted Graph Neural Networks on FPGAs for Real-Time Particle Reconstruction in High Energy Physicscited by 34
  12. 2020Fast inference of Boosted Decision Trees in FPGAs for particle physicscited by 26
1 more
  1. 2020Accelerated Charged Particle Tracking with Graph Neural Networks on FPGAscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 3; 2026: 2.

11 papers.

  1. 2025hls4ml: A Flexible, Open-Source Platform for Deep Learning Acceleration on Reconfigurable Hardwarecited by 8
  2. 2023FPGA Resource-aware Structured Pruning for Real-Time Neural Networkscited by 7
  3. 2024rule4ml: An Open-Source Tool for Resource Utilization and Latency Estimation for ML Models on FPGAcited by 3
  4. 2026JetFormer: A Scalable and Efficient Transformer for Jet Tagging from Offline Analysis to FPGA Triggerscited by 3
  5. 2023End-to-end codesign of Hessian-aware quantized neural networks for FPGAs and ASICscited by 1
  6. 2026Spiking Neural Network inference on FPGAs with hls4mlcited by 1
  7. 2023Photonic Accelerators for Image Segmentation in Autonomous Driving and Defect Detectioncited by 0
  8. 2024Exploring how deep learning decodes anomalous diffusion via Grad-CAMcited by 0
  9. 2024HPCNeuroNet: A Neuromorphic Approach Merging SNN Temporal Dynamics with Transformer Attention for FPGA-based Particle Physicscited by 0
  10. 2025Real-Time Semantic Segmentation on FPGA for Autonomous Vehicles Using LMIINet with the CGRA4ML Frameworkcited by 0
  11. 2025SparsePixels: Efficient Convolution for Sparse Data on FPGAscited by 0