# A Language Model for Particle Tracking

arXiv: https://arxiv.org/abs/2402.10239 · HTML: https://mlsanity.com/paper/2402.10239/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 14 Feb 2024.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Particle Transformer for Jet Tagging](https://mlsanity.com/paper/2202.03772/) (2022)
- [Impact of Tokenization on Language Models: An Analysis for Turkish](https://mlsanity.com/paper/2204.08832/) (2022)
- [MLPF: Efficient machine-learned particle-flow reconstruction using graph neural networks](https://mlsanity.com/paper/2101.08578/) (2021)
- [Performance of a Geometric Deep Learning Pipeline for HL-LHC Particle Tracking](https://mlsanity.com/paper/2103.06995/) (2021)
- [Towards a Computer Vision Particle Flow](https://mlsanity.com/paper/2003.08863/) (2020)
- [Machine Learning for Particle Flow Reconstruction at CMS](https://mlsanity.com/paper/2203.00330/) (2022)
- [The Tracking Machine Learning challenge : Throughput phase](https://mlsanity.com/paper/2105.01160/) (2021)

## Built on by (most cited first, 10 of 10)

- [OmniJet-α: The first cross-task foundation model for particle physics](https://mlsanity.com/paper/2403.05618/) (2024)
- [Bumblebee: Foundation Model for Particle Physics Discovery](https://mlsanity.com/paper/2412.07867/) (2024)
- [Automating High Energy Physics Data Analysis with LLM-Powered Agents](https://mlsanity.com/paper/2512.07785/) (2025)
- [Pretrained Event Classification Model for High Energy Physics Analysis](https://mlsanity.com/paper/2412.10665/) (2024)
- [Towards Foundation Models for Experimental Readout Systems Combining Discrete and Continuous Data](https://mlsanity.com/paper/2505.08736/) (2025)
- [TrackFormers: In Search of Transformer-Based Particle Tracking for the High-Luminosity LHC Era](https://mlsanity.com/paper/2407.07179/) (2024)
- [ColliderML: The First Release of an OpenDataDetector High-Luminosity Physics Benchmark Dataset](https://mlsanity.com/paper/2512.15230/) (2025)
- [Prompting Particle Physics: Tokenized Multi-modal Foundation Models for Combinatorially Many Tasks](https://mlsanity.com/paper/2609.31862/) (2026)
- [TrackSorter: A Transformer-based sorting algorithm for track finding in High Energy Physics](https://mlsanity.com/paper/2407.21290/) (2024)
- [Learning Standard Model structure from LHC data with Riemannian flow matching](https://mlsanity.com/paper/2607.16144/) (2026)

## Abstract (arXiv)

Particle tracking is crucial for almost all physics analysis programs at the Large Hadron Collider. Deep learning models are pervasively used in particle tracking related tasks. However, the current practice is to design and train one deep learning model for one task with supervised learning techniques. The trained models work well for tasks they are trained on but show no or little generalization capabilities. We propose to unify these models with a language model. In this paper, we present a tokenized detector representation that allows us to train a BERT model for particle tracking. The trained BERT model, namely TrackingBERT, offers latent detector module embedding that can be used for other tasks. This work represents the first step towards developing a foundational model for particle detector understanding.