# A Probabilistic Framework for LLM-Based Model Discovery

arXiv: https://arxiv.org/abs/2602.18266 · HTML: https://mlsanity.com/paper/2602.18266/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 76 references that are in the map.
- First posted on arXiv 20 Feb 2026.

## Builds on (most cited first, 30 of 31)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Universal Differential Equations for Scientific Machine Learning](https://mlsanity.com/paper/2001.04385/) (2020)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows](https://mlsanity.com/paper/1805.07226/) (2018)
- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [Flexible statistical inference for mechanistic models of neural dynamics](https://mlsanity.com/paper/1711.01861/) (2017)
- [LLM-SR: Scientific Equation Discovery via Programming with Large Language Models](https://mlsanity.com/paper/2404.18400/) (2024)
- [LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery](https://mlsanity.com/paper/2405.09783/) (2024)
- [Neuro-Symbolic Program Synthesis](https://mlsanity.com/paper/1611.01855/) (2016)
- [Simulation Intelligence: Towards a New Generation of Scientific Methods](https://mlsanity.com/paper/2112.03235/) (2021)
- [Elements of Sequential Monte Carlo](https://mlsanity.com/paper/1903.04797/) (2019)
- [Automated Statistical Model Discovery with Language Models](https://mlsanity.com/paper/2402.17879/) (2024)
- [AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprise](https://mlsanity.com/paper/2507.00310/) (2025)
- [Simulation-Based Inference: A Practical Guide](https://mlsanity.com/paper/2508.12939/) (2025)
- [sbi reloaded: a toolkit for simulation-based inference workflows](https://mlsanity.com/paper/2411.17337/) (2024)
- [Generalized Bayesian Inference for Scientific Simulators via Amortized Cost Estimation](https://mlsanity.com/paper/2305.15208/) (2023)
- [Generating Computational Cognitive Models using Large Language Models](https://mlsanity.com/paper/2502.00879/) (2025)
- [Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Models](https://mlsanity.com/paper/2504.17660/) (2025)
- [Large Language Bayes](https://mlsanity.com/paper/2504.14025/) (2025)
- [Language Modeling by Language Models](https://mlsanity.com/paper/2506.20249/) (2025)
- [Simultaneous identification of models and parameters of scientific simulators](https://mlsanity.com/paper/2305.15174/) (2023)
- [G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibration](https://mlsanity.com/paper/2506.09272/) (2025)

## Built on by (most cited first, 12 of 12)

- [DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking](https://mlsanity.com/paper/2605.26087/) (2026)
- [Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models](https://mlsanity.com/paper/2608.09696/) (2026)
- [Likelihood Hacking in Probabilistic Program Synthesis](https://mlsanity.com/paper/2603.24126/) (2026)
- [EvoSCM: Scientific Belief Revision Through Causal Model Evolution and Experimentation](https://mlsanity.com/paper/2609.01526/) (2026)
- [AutoStan: Autonomous Bayesian Model Improvement via Predictive Feedback](https://mlsanity.com/paper/2603.27766/) (2026)
- [Unknown Unknowns: Model Misspecification in Machine Learning for Physics](https://mlsanity.com/paper/2608.13633/) (2026)
- [Simulate, Reason, Decide: Scientific Reasoning with LLMs for Simulation-Driven Decision Making](https://mlsanity.com/paper/2606.04505/) (2026)
- [BayesEvolve: Explicit Belief States for Autonomous Scientific Discovery](https://mlsanity.com/paper/2606.30335/) (2026)
- [Program Synthesis for Simulation-Based Inference: Joint Model Selection and Parameter Estimation](https://mlsanity.com/paper/2607.17540/) (2026)
- [DASyR-LLM: Domain-Aware Symbolic Regression with LLMs for Kinetic Model Discovery](https://mlsanity.com/paper/2608.05120/) (2026)
- [Large Language Bayes Is Not Reparameterisation-Invariant](https://mlsanity.com/paper/2610.00265/) (2026)
- [Learning to Outgrow a Theory: Experimental Discovery Beyond the Initial Hypothesis Space](https://mlsanity.com/paper/2610.07627/) (2026)

## Abstract (arXiv)

Automated methods for discovering mechanistic simulator models from observational data offer a promising path toward accelerating scientific progress. Such methods often take the form of agentic-style iterative workflows that repeatedly propose and revise candidate models by imitating human discovery processes. However, existing LLM-based approaches typically implement such workflows via hand-crafted heuristic procedures, without an explicit probabilistic formulation. We recast model discovery as probabilistic inference, i.e., as sampling from an unknown distribution over mechanistic models capable of explaining the data. This perspective provides a unified way to reason about model proposal, refinement, and selection within a single inference framework. As a concrete instantiation of this view, we introduce ModelSMC, an algorithm based on Sequential Monte Carlo sampling. ModelSMC represents candidate models as particles which are iteratively proposed and refined by an LLM, and weighted using likelihood-based criteria. Experiments on real-world scientific systems illustrate that this formulation discovers models with interpretable mechanisms and improves posterior predictive checks. More broadly, this perspective provides a probabilistic lens for understanding and developing LLM-based approaches to model discovery.