mlsanityData to 7 Oct 2026

OmniJet-α: The first cross-task foundation model for particle physics

arXiv 2403.05618 ↗First posted 8 Mar 2024hep-ph cs.LG hep-excited by 41

Not on the map. Why?

Abstract (arXiv)

Foundation models are multi-dataset and multi-task machine learning methods that once pre-trained can be fine-tuned for a large variety of downstream applications. The successful development of such general-purpose models for physics data would be a major breakthrough as they could improve the achievable physics performance while at the same time drastically reduce the required amount of training time and data. We report significant progress on this challenge on several fronts. First, a comprehensive set of evaluation methods is introduced to judge the quality of an encoding from physics data into a representation suitable for the autoregressive generation of particle jets with transformer architectures (the common backbone of foundation models). These measures motivate the choice of a higher-fidelity tokenization compared to previous works. Finally, we demonstrate transfer learning between an unsupervised problem (jet generation) and a classic supervised task (jet tagging) with our new OmniJet-α model. This is the first successful transfer between two different and actively studied classes of tasks and constitutes a major step in the building of foundation models for particle physics.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 36 of its 82 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

36 of 82 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2016Layer Normalizationcited by 5,579
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
24 more
  1. 2017Deep Setscited by 1,394
  2. 2018A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decaycited by 223
  3. 2022Particle Transformer for Jet Taggingcited by 95
  4. 2018Energy Flow Networks: Deep Sets for Particle Jetscited by 83
  5. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  6. 2023Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networkscited by 80
  7. 2017Accelerating Science with Generative Adversarial Networks: An Application to 3D Particle Showers in Multi-Layer Calorimeterscited by 55
  8. 2021Machine Learning in the Search for New Fundamental Physicscited by 50
  9. 2017Learning Particle Physics by Example: Location-Aware Generative Adversarial Networks for Physics Synthesiscited by 46
  10. 2024Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Modelscited by 44
  11. 2023CaloClouds: Fast Geometry-Independent Highly-Granular Calorimeter Simulationcited by 33
  12. 2023Ultra-High-Resolution Detector Simulation with Intra-Event Aware GAN and Self-Supervised Relational Reasoningcited by 25
  13. 2023EPiC-GAN: Equivariant Point Cloud Generation for Particle Jetscited by 23
  14. 2023CaloClouds II: Ultra-Fast Geometry-Independent Highly-Granular Calorimeter Simulationcited by 20
  15. 2023Deep Generative Models for Detector Signature Simulation: A Taxonomic Reviewcited by 18
  16. 2024Finetuning Foundation Models for Joint Analysis Optimizationcited by 18
  17. 2022New directions for surrogate models and differentiable programming for High Energy Physics detector simulationcited by 16
  18. 2023Anomalies, Representations, and Self-Supervisioncited by 10
  19. 2024A Language Model for Particle Trackingcited by 10
  20. 2022Neural Embedding: Learning the Embedding of the Manifold of Physics Datacited by 7
  21. 2023Flow Matching Beyond Kinematics: Generating Jets with Particle-ID and Trajectory Displacement Informationcited by 7
  22. 2023Generalizing to new geometries with Geometry-Aware Autoregressive Models (GAAMs) for fast calorimeter simulationcited by 5
  23. 2021Shared Data and Algorithms for Deep Learning in Fundamental Physicscited by 4
  24. 2023Improving the performance of weak supervision searches using transfer and meta-learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 16; 2026: 18.

41 papers.

Sort:

By area:

  1. 2024Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Modelscited by 21
  2. 2024PAPERCLIP: Associating Astronomical Observations and Natural Language with Multi-Modal Modelscited by 15
  3. 2024Calo-VQ: Vector-Quantized Two-Stage Generative Model in Calorimeter Simulationcited by 15
  4. 2025Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Modelscited by 14
  5. 2025OmniJet-αC: Learning point cloud calorimeter simulations using generative transformerscited by 14
  6. 2024Convolutional L2LFlows: Generating Accurate Showers in Highly Granular Calorimeters Using Convolutional Normalizing Flowscited by 13
  7. 2026Neural Scaling Laws for Boosted Jet Taggingcited by 11
  8. 2025Foundation models for high-energy physicscited by 10
  9. 2024FEET: A Framework for Evaluating Embedding Techniquescited by 8
  10. 2024Pretrained Event Classification Model for High Energy Physics Analysiscited by 8
  11. 2025Fine-tuning machine-learned particle-flow reconstruction for new detector geometries in future colliderscited by 8
  12. 2026EveNet: A Foundation Model for Particle Collision Data Analysiscited by 8
29 more
  1. 2025RINO: Renormalization Group Invariance with No Labelscited by 6
  2. 2025Towards Foundation Models for Experimental Readout Systems Combining Discrete and Continuous Datacited by 5
  3. 2025Science Is Falling Behind the Frontier: Foundation Model Adoption Across Half a Million Paperscited by 5
  4. 2025SURFing to the Fundamental Limit of Jet Taggingcited by 3
  5. 2025Enhancing next token prediction based pre-training for jet foundation modelscited by 3
  6. 2026Cross-Domain Transfer with Particle Physics Foundation Models: From Jets to Neutrino Interactionscited by 3
  7. 2024Extrapolating Jet Radiation with Autoregressive Transformerscited by 2
  8. 2025Stable and Interpretable Jet Physics with IRC-Safe Equivariant Feature Extractioncited by 2
  9. 2026Towards foundation-style models for energy-frontier heterogeneous neutrino detectors via self-supervised pre-trainingcited by 2
  10. 2026Generative Models and Statistical Validationcited by 2
  11. 2026JetParticle-JEPA: An Efficient Self-Supervised Representation Learning method for Jet Tagging in High-Energy Physicscited by 2
  12. 2025BitHEP -- The Limits of Low-Precision ML in HEPcited by 1
  13. 2025Grounding Intelligence in Movementcited by 1
  14. 2025Double Descent and Overparameterization in Particle Physics Datacited by 1
  15. 2025Latent Representation Learning in Heavy-Ion Collisions with MaskPoint Transformercited by 1
  16. 2026jBOT: Semantic Jet Representation Clustering Emerges from Self-Distillationcited by 1
  17. 2026Application of a Mixture of Experts-based Foundation Model to the GlueX DIRC Detectorcited by 1
  18. 2026Machine-learned particle flow as a foundation model for collider physicscited by 1
  19. 2026Towards Engineering Scaling Laws with Pretraining Data Compositioncited by 1
  20. 2026A Lightweight Foundation Model for Collider Physics with Multi-Domain Adaptationcited by 1
  21. 2026Pairton: Iterative Reconstruction of Short-Lived Particlescited by 1
  22. 2026Prompting Particle Physics: Tokenized Multi-modal Foundation Models for Combinatorially Many Taskscited by 1
  23. 2025PLAID: A Unified Data Model for Machine Learning on Heterogeneous Physics Simulationscited by 0
  24. 2025NuBench: An Open Benchmark for Deep Learning-Based Event Reconstruction in Neutrino Telescopescited by 0
  25. 2026Learning Standard Model structure from LHC data with Riemannian flow matchingcited by 0
  26. 2026"Transforming" LHCb: self-supervised maps of heavy-flavour decayscited by 0
  27. 2026Leveraging Industrial Foundation Models at the Edge of Particle Physics Detectors via Distillation Learning and Hardware Co-designcited by 0
  28. 2026Machine learning for the LHC physics program: a 2025-2026 stocktakecited by 0
  29. 2026Scaling Collider Event Generation with Residual-Quantized Tokenscited by 0