mlsanityData to 14 Oct 2026

Bumblebee: Foundation Model for Particle Physics Discovery

arXiv 2412.07867 ↗First posted 10 Dec 2024hep-ex cs.LG hep-phcited by 12

Not on the map. Why?

Abstract (arXiv)

Bumblebee is a foundation model for particle physics discovery, inspired by BERT. By removing positional encodings and embedding particle 4-vectors, Bumblebee captures both generator- and reconstruction-level information while ensuring sequence-order invariance. Pre-trained on a masked task, it improves dileptonic top quark reconstruction resolution by 10-20% and excels in downstream tasks, including toponium discrimination (AUROC 0.877) and initial state classification (AUROC 0.625). The flexibility of Bumblebee makes it suitable for a wide range of particle physics applications, especially the discovery of new particles.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 9 of its 43 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

9 of 43 references.

References · 9

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  8. 2024A Language Model for Particle Trackingcited by 10
  9. 2024VIS-MAE: An Efficient Self-supervised Learning Approach on Medical Image Segmentation and Classificationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 5.

12 papers.

  1. 2025Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Modelscited by 14
  2. 2025OmniJet-αC: Learning point cloud calorimeter simulations using generative transformerscited by 14
  3. 2025Foundation models for high-energy physicscited by 10
  4. 2024Pretrained Event Classification Model for High Energy Physics Analysiscited by 8
  5. 2025Towards Foundation Models for Experimental Readout Systems Combining Discrete and Continuous Datacited by 5
  6. 2025Enhancing next token prediction based pre-training for jet foundation modelscited by 3
  7. 2026A universal vision transformer for fast calorimeter simulationscited by 2
  8. 2026jBOT: Semantic Jet Representation Clustering Emerges from Self-Distillationcited by 2
  9. 2026Towards Engineering Scaling Laws with Pretraining Data Compositioncited by 2
  10. 2026Prompting Particle Physics: Tokenized Multi-modal Foundation Models for Combinatorially Many Taskscited by 2
  11. 2025BitHEP -- The Limits of Low-Precision ML in HEPcited by 1
  12. 2026Application of a Mixture of Experts-based Foundation Model to the GlueX DIRC Detectorcited by 1