# Simulation as Supervision: Mechanistic Pretraining for Scientific Discovery

arXiv: https://arxiv.org/abs/2507.08977 · HTML: https://mlsanity.com/paper/2507.08977/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 48 references that are in the map.
- First posted on arXiv 11 Jul 2025.

## Builds on (most cited first, 14 of 14)

- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Universal Differential Equations for Scientific Machine Learning](https://mlsanity.com/paper/2001.04385/) (2020)
- [Benchmarking Simulation-Based Inference](https://mlsanity.com/paper/2101.04653/) (2021)
- [All-in-one simulation-based inference](https://mlsanity.com/paper/2404.09636/) (2024)
- [TimePFN: Effective Multivariate Time Series Forecasting with Synthetic Data](https://mlsanity.com/paper/2502.16294/) (2025)
- [EINNs: Epidemiologically-informed Neural Networks](https://mlsanity.com/paper/2202.10446/) (2022)
- [Generalized Laplacian Positional Encoding for Graph Representation Learning](https://mlsanity.com/paper/2210.15956/) (2022)

## Built on by (most cited first, 3 of 3)

- [Mantis: A Foundation Model for Mechanistic Disease Forecasting](https://mlsanity.com/paper/2508.12260/) (2025)
- [Learning From Simulators: A Theory of Simulation-Grounded Learning](https://mlsanity.com/paper/2509.18990/) (2025)
- [In-Context Learning Under Regime Change](https://mlsanity.com/paper/2604.16988/) (2026)

## Abstract (arXiv)

Scientific modeling faces a tradeoff between the interpretability of mechanistic theory and the predictive power of machine learning. While existing hybrid approaches have made progress by incorporating domain knowledge into machine learning methods as functional constraints, they can be limited by a reliance on precise mathematical specifications. When the underlying equations are partially unknown or misspecified, enforcing rigid constraints can introduce bias and hinder a model's ability to learn from data. We introduce Simulation-Grounded Neural Networks (SGNNs), a framework that incorporates scientific theory by using mechanistic simulations as training data for neural networks. By pretraining on diverse synthetic corpora that span multiple model structures and realistic observational noise, SGNNs internalize the underlying dynamics of a system as a structural prior. We evaluated SGNNs across multiple disciplines, including epidemiology, ecology, social science, and chemistry. In forecasting tasks, SGNNs outperformed both standard data-driven baselines and physics-constrained hybrid models. They nearly tripled the forecasting skill of the average CDC models in COVID-19 mortality forecasts and accurately forecasted high-dimensional ecological systems. SGNNs demonstrated robustness to model misspecification, performing well even when trained on data with incorrect assumptions. Our framework also introduces back-to-simulation attribution, a method for mechanistic interpretability that explains real-world dynamics by identifying their most similar counterparts within the simulated corpus. By unifying these techniques into a single framework, we demonstrate that diverse mechanistic simulations can serve as effective training data for robust scientific inference.