# Pre-Training for Simulation-Based Science: A Study on Jet Foundation Model Training Objectives

arXiv: https://arxiv.org/abs/2606.14870 · HTML: https://mlsanity.com/paper/2606.14870/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Flow matching, time series and science, one of 14 directions in Diffusion models.
- Builds on 7 of its 20 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Placed in

- Diffusion models / [Flow matching, time series and science](https://mlsanity.com/diffusion/flow-matching-time-series-and-science/)

## Builds on (most cited first, 7 of 7)

- [UMA: A Family of Universal Models for Atoms](https://mlsanity.com/paper/2506.23971/) (2025)
- [Walrus: A Cross-Domain Foundation Model for Continuum Dynamics](https://mlsanity.com/paper/2511.15684/) (2025)
- [HEP-JEPA: A foundation model for collider physics using joint embedding predictive architecture](https://mlsanity.com/paper/2502.03933/) (2025)
- [Neural Scaling Laws for Boosted Jet Tagging](https://mlsanity.com/paper/2602.15781/) (2026)
- [Zatom-1: Towards a Multimodal Foundation Model for 3D Molecules and Materials](https://mlsanity.com/paper/2602.22251/) (2026)
- [Neural Scaling Laws for Jet Generation](https://mlsanity.com/paper/2605.28940/) (2026)
- [Cross-Domain Transfer with Particle Physics Foundation Models: From Jets to Neutrino Interactions](https://mlsanity.com/paper/2604.12364/) (2026)

## Built on by (most cited first, 1 of 1)

- [Searching for BSM Experimental Signatures with Large Lagrangian Models](https://mlsanity.com/paper/2609.38309/) (2026)

## Abstract (arXiv)

Foundation models (FMs) trained on large datasets and fine-tuned on downstream tasks have emerged as a powerful paradigm in AI for science. Industrial FMs are typically trained using self-supervision with masking due to the lack of labels. In many scientific domains, accurate simulations are plentiful and facilitate large, labeled datasets. This opens up new possibilities for pre-training. We present a systematic comparison of pre-training methods using the OmniLearned High Energy Physics FM framework. We test supervised classification, flow-matching generation, and self-supervised masked particle modeling. All models are pre-trained on the JetClass dataset and fine-tuned on two representative downstream tasks, top jet classification and JetNet conditional generation. Among other observations, for classification tasks, we find that pure classifier pre-training is optimal when downstream labels and model capacity are plentiful, but combining it with self-supervised masked particle modeling (MPM) is uniquely powerful in the low-finetuning label regime. Flow matching-based generative pre-training seems to provide little benefit for downstream classification, and interestingly, for downstream generation, we find that flow matching must be in the pre-training objective to see a significant finetuning advantage, hinting at the orthogonality of classification and generation tasks. That is, for a model to transfer to both generative and classification downstream tasks, it must be pre-trained on both. This study provides a template for controlled scaling analysis of pre-training objectives for foundation models in simulation-based sciences.