# Ultra-low latency recurrent neural network inference on FPGAs for physics applications with hls4ml

arXiv: https://arxiv.org/abs/2207.00559 · HTML: https://mlsanity.com/paper/2207.00559/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 42 references that are in the map.
- First posted on arXiv 1 Jul 2022.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Fast inference of deep neural networks in FPGAs for particle physics](https://mlsanity.com/paper/1804.06913/) (2018)
- [Compressing deep neural networks on FPGAs to binary and ternary precision with HLS4ML](https://mlsanity.com/paper/2003.06308/) (2020)
- [Fast convolutional neural networks on FPGAs with hls4ml](https://mlsanity.com/paper/2101.05108/) (2021)
- [A reconfigurable neural network ASIC for detector front-end data compression at the HL-LHC](https://mlsanity.com/paper/2105.01683/) (2021)
- [Long Short-Term Memory (LSTM) networks with jet constituents for boosted top tagging at the LHC](https://mlsanity.com/paper/1711.09059/) (2017)
- [Accelerating Recurrent Neural Networks for Gravitational Wave Experiments](https://mlsanity.com/paper/2106.14089/) (2021)
- [FINN-L: Library Extensions and Design Trade-off Analysis for Variable Precision LSTM Networks on FPGAs](https://mlsanity.com/paper/1807.04093/) (2018)
- [FPGAs-as-a-Service Toolkit (FaaST)](https://mlsanity.com/paper/2010.08556/) (2020)
- [Recurrent Neural Networks Hardware Implementation on FPGA](https://mlsanity.com/paper/1511.05552/) (2015)
- [Jet Charge and Machine Learning](https://mlsanity.com/paper/1803.08066/) (2018)
- [Development of a Vertex Finding Algorithm using Recurrent Neural Network](https://mlsanity.com/paper/2101.11906/) (2021)
- [Sequence-based Machine Learning Models in Jet Physics](https://mlsanity.com/paper/2102.06128/) (2021)
- [FPGA-Based Low-Power Speech Recognition with Recurrent Neural Networks](https://mlsanity.com/paper/1610.00552/) (2016)

## Built on by (most cited first, 14 of 14)

- [hls4ml: A Flexible, Open-Source Platform for Deep Learning Acceleration on Reconfigurable Hardware](https://mlsanity.com/paper/2512.01463/) (2025)
- [On-Sensor Data Filtering using Neuromorphic Computing for High Energy Physics Experiments](https://mlsanity.com/paper/2307.11242/) (2023)
- [FPGA Resource-aware Structured Pruning for Real-Time Neural Networks](https://mlsanity.com/paper/2308.05170/) (2023)
- [Ultra Fast Transformers on FPGAs for Particle Physics Experiments](https://mlsanity.com/paper/2402.01047/) (2024)
- [Low Latency Transformer Inference on FPGAs for Physics Applications with hls4ml](https://mlsanity.com/paper/2409.05207/) (2024)
- [rule4ml: An Open-Source Tool for Resource Utilization and Latency Estimation for ML Models on FPGA](https://mlsanity.com/paper/2408.05314/) (2024)
- [JetFormer: A Scalable and Efficient Transformer for Jet Tagging from Offline Analysis to FPGA Triggers](https://mlsanity.com/paper/2601.17215/) (2026)
- [BitHEP -- The Limits of Low-Precision ML in HEP](https://mlsanity.com/paper/2504.03387/) (2025)
- [Scintillation pulse characterization with spectrum-inspired temporal neural networks: case studies on particle detector signals](https://mlsanity.com/paper/2410.07267/) (2024)
- [HPCNeuroNet: A Neuromorphic Approach Merging SNN Temporal Dynamics with Transformer Attention for FPGA-based Particle Physics](https://mlsanity.com/paper/2412.17571/) (2024)
- [FINN-GL: Generalized Mixed-Precision Extensions for FPGA-Accelerated LSTMs](https://mlsanity.com/paper/2506.20810/) (2025)
- [Neural Network Acceleration on MPSoC board: Integrating SLAC's SNL, Rogue Software and Auto-SNL](https://mlsanity.com/paper/2508.21739/) (2025)
- [SparsePixels: Efficient Convolution for Sparse Data on FPGAs](https://mlsanity.com/paper/2512.06208/) (2025)
- [Real-Time Plasma State Prediction via FPGA-Accelerated Quantized Recurrent Probabilistic Neural Networks](https://mlsanity.com/paper/2609.23141/) (2026)

## Abstract (arXiv)

Recurrent neural networks have been shown to be effective architectures for many tasks in high energy physics, and thus have been widely adopted. Their use in low-latency environments has, however, been limited as a result of the difficulties of implementing recurrent architectures on field-programmable gate arrays (FPGAs). In this paper we present an implementation of two types of recurrent neural network layers -- long short-term memory and gated recurrent unit -- within the hls4ml framework. We demonstrate that our implementation is capable of producing effective designs for both small and large models, and can be customized to meet specific design requirements for inference latencies and FPGA resources. We show the performance and synthesized designs for multiple neural networks, many of which are trained specifically for jet identification tasks at the CERN Large Hadron Collider.