# Towards Engineering Scaling Laws with Pretraining Data Composition

arXiv: https://arxiv.org/abs/2606.19781 · HTML: https://mlsanity.com/paper/2606.19781/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 18 Jun 2026.

## Builds on (most cited first, 27 of 27)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [Particle Transformer for Jet Tagging](https://mlsanity.com/paper/2202.03772/) (2022)
- [A Neural Scaling Law from the Dimension of the Data Manifold](https://mlsanity.com/paper/2004.10802/) (2020)
- [Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models](https://mlsanity.com/paper/2401.13537/) (2024)
- [OmniJet-α: The first cross-task foundation model for particle physics](https://mlsanity.com/paper/2403.05618/) (2024)
- [Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models](https://mlsanity.com/paper/2403.07066/) (2024)
- [HEP-JEPA: A foundation model for collider physics using joint embedding predictive architecture](https://mlsanity.com/paper/2502.03933/) (2025)
- [Learning Symmetry-Independent Jet Representations via Jet-Based Joint Embedding Predictive Architecture](https://mlsanity.com/paper/2412.05333/) (2024)
- [Bumblebee: Foundation Model for Particle Physics Discovery](https://mlsanity.com/paper/2412.07867/) (2024)
- [Neural Scaling Laws for Boosted Jet Tagging](https://mlsanity.com/paper/2602.15781/) (2026)
- [Foundation models for high-energy physics](https://mlsanity.com/paper/2509.21434/) (2025)
- [EveNet: A Foundation Model for Particle Collision Data Analysis](https://mlsanity.com/paper/2601.17126/) (2026)
- [Pretrained Event Classification Model for High Energy Physics Analysis](https://mlsanity.com/paper/2412.10665/) (2024)
- [Particle Trajectory Representation Learning with Masked Point Modeling](https://mlsanity.com/paper/2502.02558/) (2025)
- [Scaling Laws in Jet Classification](https://mlsanity.com/paper/2312.02264/) (2023)
- [FM4NPP: A Scaling Foundation Model for Nuclear and Particle Physics](https://mlsanity.com/paper/2508.14087/) (2025)
- [Scaling laws for amplitude surrogates](https://mlsanity.com/paper/2601.13308/) (2026)
- [Neural Scaling Laws for Jet Generation](https://mlsanity.com/paper/2605.28940/) (2026)
- [Panda: Self-distillation of Reusable Sensor-level Representations for High Energy Physics](https://mlsanity.com/paper/2512.01324/) (2025)

## Built on by (most cited first, 2 of 2)

- [Prompting Particle Physics: Tokenized Multi-modal Foundation Models for Combinatorially Many Tasks](https://mlsanity.com/paper/2609.31862/) (2026)
- [How to scale your HEP ML models: A recipe for robust architecture comparisons at scale](https://mlsanity.com/paper/2610.06784/) (2026)

## Abstract (arXiv)

Neural scaling laws describe how model performance improves as a power law in compute, model size, and dataset size. While well-established for large language models, these relationships are emerging for large models in particle physics. As with language, empirical studies show that the performance scales as a power law. However, unlike natural language or image domains, fundamental physics has high-fidelity simulators that produce synthetic data cheaply. This favors scaling regimes where additional data is cheaper than additional parameters, and allows the pretraining dataset itself to be engineered to influence the scaling. For the task of classifying hadronic jets produced in collisions of high-energy particle beams, we show that the scaling behavior can be engineered towards requiring more data rather than larger models by inclusion of pretraining data which is more diverse and better aligned with the downstream classification task.