# Real-time semantic segmentation on FPGAs for autonomous vehicles with hls4ml

arXiv: https://arxiv.org/abs/2205.07690 · HTML: https://mlsanity.com/paper/2205.07690/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 16 May 2022.

## Builds on (most cited first, 13 of 13)

- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1606.02147/) (2016)
- [Fast inference of deep neural networks in FPGAs for particle physics](https://mlsanity.com/paper/1804.06913/) (2018)
- [Benchmarking TinyML Systems: Challenges and Direction](https://mlsanity.com/paper/2003.04821/) (2020)
- [Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors](https://mlsanity.com/paper/2006.10159/) (2020)
- [hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devices](https://mlsanity.com/paper/2103.05579/) (2021)
- [Compressing deep neural networks on FPGAs to binary and ternary precision with HLS4ML](https://mlsanity.com/paper/2003.06308/) (2020)
- [Fast convolutional neural networks on FPGAs with hls4ml](https://mlsanity.com/paper/2101.05108/) (2021)
- [Distance-Weighted Graph Neural Networks on FPGAs for Real-Time Particle Reconstruction in High Energy Physics](https://mlsanity.com/paper/2008.03601/) (2020)
- [Fast inference of Boosted Decision Trees in FPGAs for particle physics](https://mlsanity.com/paper/2002.02534/) (2020)
- [Accelerated Charged Particle Tracking with Graph Neural Networks on FPGAs](https://mlsanity.com/paper/2012.01563/) (2020)

## Built on by (most cited first, 11 of 11)

- [hls4ml: A Flexible, Open-Source Platform for Deep Learning Acceleration on Reconfigurable Hardware](https://mlsanity.com/paper/2512.01463/) (2025)
- [FPGA Resource-aware Structured Pruning for Real-Time Neural Networks](https://mlsanity.com/paper/2308.05170/) (2023)
- [rule4ml: An Open-Source Tool for Resource Utilization and Latency Estimation for ML Models on FPGA](https://mlsanity.com/paper/2408.05314/) (2024)
- [JetFormer: A Scalable and Efficient Transformer for Jet Tagging from Offline Analysis to FPGA Triggers](https://mlsanity.com/paper/2601.17215/) (2026)
- [End-to-end codesign of Hessian-aware quantized neural networks for FPGAs and ASICs](https://mlsanity.com/paper/2304.06745/) (2023)
- [Spiking Neural Network inference on FPGAs with hls4ml](https://mlsanity.com/paper/2606.10008/) (2026)
- [Photonic Accelerators for Image Segmentation in Autonomous Driving and Defect Detection](https://mlsanity.com/paper/2309.16783/) (2023)
- [Exploring how deep learning decodes anomalous diffusion via Grad-CAM](https://mlsanity.com/paper/2410.16345/) (2024)
- [HPCNeuroNet: A Neuromorphic Approach Merging SNN Temporal Dynamics with Transformer Attention for FPGA-based Particle Physics](https://mlsanity.com/paper/2412.17571/) (2024)
- [Real-Time Semantic Segmentation on FPGA for Autonomous Vehicles Using LMIINet with the CGRA4ML Framework](https://mlsanity.com/paper/2510.22243/) (2025)
- [SparsePixels: Efficient Convolution for Sparse Data on FPGAs](https://mlsanity.com/paper/2512.06208/) (2025)

## Abstract (arXiv)

In this paper, we investigate how field programmable gate arrays can serve as hardware accelerators for real-time semantic segmentation tasks relevant for autonomous driving. Considering compressed versions of the ENet convolutional neural network architecture, we demonstrate a fully-on-chip deployment with a latency of 4.9 ms per image, using less than 30% of the available resources on a Xilinx ZCU102 evaluation board. The latency is reduced to 3 ms per image when increasing the batch size to ten, corresponding to the use case where the autonomous vehicle receives inputs from multiple cameras simultaneously. We show, through aggressive filter reduction and heterogeneous quantization-aware training, and an optimized implementation of convolutional layers, that the power consumption and resource utilization can be significantly reduced while maintaining accuracy on the Cityscapes dataset.