mlsanityData to 7 Oct 2026

Ultra Fast Transformers on FPGAs for Particle Physics Experiments

arXiv 2402.01047 ↗First posted 1 Feb 2024cs.LG cs.AR hep-excited by 6

Not on the map. Why?

Abstract (arXiv)

This work introduces a highly efficient implementation of the transformer architecture on a Field-Programmable Gate Array (FPGA) by using the texttthls4ml tool. Given the demonstrated effectiveness of transformer models in addressing a wide range of problems, their application in experimental triggers within particle physics becomes a subject of significant interest. In this work, we have implemented critical components of a transformer model, such as multi-head attention and softmax layers. To evaluate the effectiveness of our implementation, we have focused on a particle physics jet flavor tagging problem, employing a public dataset. We recorded latency under 2 μs on the Xilinx UltraScale+ FPGA, which is compatible with hardware trigger requirements at the CERN Large Hadron Collider experiments.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 12 of its 24 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

12 of 24 references.

References · 12

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018Fast inference of deep neural networks in FPGAs for particle physicscited by 132
  3. 2022Particle Transformer for Jet Taggingcited by 95
  4. 2021hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devicescited by 61
  5. 2020FTRANS: Energy-Efficient Acceleration of Transformers using FPGAcited by 52
  6. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 46
  7. 2021Fast convolutional neural networks on FPGAs with hls4mlcited by 42
  8. 2021Point Cloud Transformers applied to Collider Physicscited by 23
  9. 2021SPANet: Generalized Permutationless Set Assignment for Particle Physics using Symmetry Preserving Attentioncited by 18
  10. 2021Graph Neural Networks for Charged Particle Tracking on FPGAscited by 17
  11. 2022Ultra-low latency recurrent neural network inference on FPGAs for physics applications with hls4mlcited by 14
  12. 2021Accelerating Recurrent Neural Networks for Gravitational Wave Experimentscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 2.

6 papers.

  1. 2025Sub-microsecond Transformers for Jet Tagging on FPGAscited by 6
  2. 2024Ultra-low latency quantum-inspired machine learning predictors implemented on FPGAcited by 4
  3. 2026JetFormer: A Scalable and Efficient Transformer for Jet Tagging from Offline Analysis to FPGA Triggerscited by 3
  4. 2026Towards Tensor Network Models for Low-Latency Jet Tagging on FPGAscited by 2
  5. 2024ProTEA: Programmable Transformer Encoder Acceleration on FPGAcited by 1
  6. 2024FAMOUS: Flexible Accelerator for the Attention Mechanism of Transformer on UltraScale+ FPGAscited by 1