# Ultra Fast Transformers on FPGAs for Particle Physics Experiments

arXiv: https://arxiv.org/abs/2402.01047 · HTML: https://mlsanity.com/paper/2402.01047/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 1 Feb 2024.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Fast inference of deep neural networks in FPGAs for particle physics](https://mlsanity.com/paper/1804.06913/) (2018)
- [Particle Transformer for Jet Tagging](https://mlsanity.com/paper/2202.03772/) (2022)
- [hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devices](https://mlsanity.com/paper/2103.05579/) (2021)
- [FTRANS: Energy-Efficient Acceleration of Transformers using FPGA](https://mlsanity.com/paper/2007.08563/) (2020)
- [DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation](https://mlsanity.com/paper/2209.10797/) (2022)
- [Fast convolutional neural networks on FPGAs with hls4ml](https://mlsanity.com/paper/2101.05108/) (2021)
- [Point Cloud Transformers applied to Collider Physics](https://mlsanity.com/paper/2102.05073/) (2021)
- [SPANet: Generalized Permutationless Set Assignment for Particle Physics using Symmetry Preserving Attention](https://mlsanity.com/paper/2106.03898/) (2021)
- [Graph Neural Networks for Charged Particle Tracking on FPGAs](https://mlsanity.com/paper/2112.02048/) (2021)
- [Ultra-low latency recurrent neural network inference on FPGAs for physics applications with hls4ml](https://mlsanity.com/paper/2207.00559/) (2022)
- [Accelerating Recurrent Neural Networks for Gravitational Wave Experiments](https://mlsanity.com/paper/2106.14089/) (2021)

## Built on by (most cited first, 6 of 6)

- [Sub-microsecond Transformers for Jet Tagging on FPGAs](https://mlsanity.com/paper/2510.24784/) (2025)
- [Ultra-low latency quantum-inspired machine learning predictors implemented on FPGA](https://mlsanity.com/paper/2409.16075/) (2024)
- [JetFormer: A Scalable and Efficient Transformer for Jet Tagging from Offline Analysis to FPGA Triggers](https://mlsanity.com/paper/2601.17215/) (2026)
- [Towards Tensor Network Models for Low-Latency Jet Tagging on FPGAs](https://mlsanity.com/paper/2601.10801/) (2026)
- [ProTEA: Programmable Transformer Encoder Acceleration on FPGA](https://mlsanity.com/paper/2409.13975/) (2024)
- [FAMOUS: Flexible Accelerator for the Attention Mechanism of Transformer on UltraScale+ FPGAs](https://mlsanity.com/paper/2409.14023/) (2024)

## Abstract (arXiv)

This work introduces a highly efficient implementation of the transformer architecture on a Field-Programmable Gate Array (FPGA) by using the texttthls4ml tool. Given the demonstrated effectiveness of transformer models in addressing a wide range of problems, their application in experimental triggers within particle physics becomes a subject of significant interest. In this work, we have implemented critical components of a transformer model, such as multi-head attention and softmax layers. To evaluate the effectiveness of our implementation, we have focused on a particle physics jet flavor tagging problem, employing a public dataset. We recorded latency under 2 μs on the Xilinx UltraScale+ FPGA, which is compatible with hardware trigger requirements at the CERN Large Hadron Collider experiments.