# Towards foundation-style models for energy-frontier heterogeneous neutrino detectors via self-supervised pre-training

arXiv: https://arxiv.org/abs/2604.07037 · HTML: https://mlsanity.com/paper/2604.07037/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 59 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [GraphMAE: Self-Supervised Masked Graph Autoencoders](https://mlsanity.com/paper/2205.10803/) (2022)
- [Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors](https://mlsanity.com/paper/2006.10159/) (2020)
- [Machine Learning in the Search for New Fundamental Physics](https://mlsanity.com/paper/2112.03769/) (2021)
- [Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models](https://mlsanity.com/paper/2401.13537/) (2024)
- [OmniJet-α: The first cross-task foundation model for particle physics](https://mlsanity.com/paper/2403.05618/) (2024)
- [A Convolutional Neural Network Neutrino Event Classifier](https://mlsanity.com/paper/1604.01444/) (2016)
- [Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models](https://mlsanity.com/paper/2403.07066/) (2024)
- [Quantum anomaly detection in the latent space of proton collision events at the LHC](https://mlsanity.com/paper/2301.10780/) (2023)
- [Finetuning Foundation Models for Joint Analysis Optimization](https://mlsanity.com/paper/2401.13536/) (2024)
- [Scalable Deep Convolutional Neural Networks for Sparse, Locally Dense Liquid Argon Time Projection Chamber Data](https://mlsanity.com/paper/1903.05663/) (2019)
- [Particle Trajectory Representation Learning with Masked Point Modeling](https://mlsanity.com/paper/2502.02558/) (2025)
- [PILArNet: Public Dataset for Particle Imaging Liquid Argon Detectors in High Energy Physics](https://mlsanity.com/paper/2006.01993/) (2020)
- [Improved particle-flow event reconstruction with scalable neural networks for current and future particle detectors](https://mlsanity.com/paper/2309.06782/) (2023)
- [Adapting Vision-Language Models for Neutrino Event Classification in High-Energy Physics](https://mlsanity.com/paper/2509.08461/) (2025)
- [Artificial intelligence for improved fitting of trajectories of elementary particles in inhomogeneous dense materials immersed in a magnetic field](https://mlsanity.com/paper/2211.04890/) (2022)
- [Transfer Learning for Neutrino Scattering: Domain Adaptation with GANs](https://mlsanity.com/paper/2508.12987/) (2025)

## Built on by (most cited first, 2 of 2)

- [Panda Diplomacy: Foundation Model Pre-training across Particle Imaging Detectors for High Energy and Nuclear Physics](https://mlsanity.com/paper/2609.00611/) (2026)
- [Machine learning for the LHC physics program: a 2025-2026 stocktake](https://mlsanity.com/paper/2609.32874/) (2026)

## Abstract (arXiv)

Accelerator-based neutrino physics is entering an energy-frontier regime in which interactions reach the TeV scale and produce exceptionally dense, overlapping detector signatures. In this regime, event interpretation becomes impractical for conventional reconstruction approaches, particularly when labelled data are scarce and the analysis spans diverse downstream objectives. We present a sparse ViT framework for learning reusable representations from heterogeneous detector data. Self-supervised pre-training combines masked autoencoder reconstruction with relational voxel-level objectives for hierarchy, ghost and particle identification, and the resulting shared encoder is then jointly fine-tuned across classification and regression tasks. Evaluated on simulated events from the proposed FASERCal concept at the LHC, we find that pre-training consistently improves neutrino flavour and charm-quark identification, momentum regression, and vertex reconstruction over training from scratch, with the addition of relational objectives yielding further gains in the most topologically complex channels. Interpretability analyses further show that pre-training yields a more structured latent space, while detector-subsystem ablations recover physically plausible channel-dependent roles for the heterogeneous inputs. A data-efficiency study shows that, with roughly 103 labelled events, the pre-trained encoder already matches the flavour-classification performance of a randomly initialised model trained on an order of magnitude more data. The learned representations also transfer effectively to publicly available benchmarks spanning different detector technologies and energy scales, matching or exceeding published baselines. These results support self-supervised pre-training on multimodal detector data as a scalable route towards reusable representations for neutrino and particle-detector analysis.