# All Emulators are Wrong, Many are Useful, and Some are More Useful Than Others: A Reproducible Comparison of Computer Model Surrogates

arXiv: https://arxiv.org/abs/2512.09060 · HTML: https://mlsanity.com/paper/2512.09060/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 161 references that are in the map.
- First posted on arXiv 9 Dec 2025.

## Builds on (most cited first, 30 of 40)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [When Gaussian Process Meets Big Data: A Review of Scalable GPs](https://mlsanity.com/paper/1807.01065/) (2018)
- [NGBoost: Natural Gradient Boosting for Probabilistic Prediction](https://mlsanity.com/paper/1910.03225/) (2019)
- [Distributed Gaussian Processes](https://mlsanity.com/paper/1502.02843/) (2015)
- [Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification](https://mlsanity.com/paper/1802.00850/) (2018)
- [Exploring the Whole Rashomon Set of Sparse Decision Trees](https://mlsanity.com/paper/2209.08040/) (2022)
- [A general framework for Vecchia approximations of Gaussian processes](https://mlsanity.com/paper/1708.06302/) (2017)
- [Local Gaussian process approximation for large computer experiments](https://mlsanity.com/paper/1303.0383/) (2013)
- [Amazing Things Come From Having Many Good Models](https://mlsanity.com/paper/2407.04846/) (2024)
- [A Case Study Competition Among Methods for Analyzing Large Spatial Data](https://mlsanity.com/paper/1710.05013/) (2017)
- [JANA: Jointly Amortized Neural Approximation of Complex Bayesian Models](https://mlsanity.com/paper/2302.09125/) (2023)
- [Practical heteroskedastic Gaussian process modeling for large simulation experiments](https://mlsanity.com/paper/1611.05902/) (2016)
- [Active Learning for Deep Gaussian Process Surrogates](https://mlsanity.com/paper/2012.08015/) (2020)
- [Polynomial-Chaos-based Kriging](https://mlsanity.com/paper/1502.03939/) (2015)
- [Vecchia approximations of Gaussian-process predictions](https://mlsanity.com/paper/1805.03309/) (2018)
- [Extending classical surrogate modelling to high-dimensions through supervised dimensionality reduction: a data-driven approach](https://mlsanity.com/paper/1812.06309/) (2018)
- [Sequential Learning of Active Subspaces](https://mlsanity.com/paper/1907.11572/) (2019)
- [Scaled Vecchia approximation for fast computer-model emulation](https://mlsanity.com/paper/2005.00386/) (2020)
- [Nonstationary Gaussian Process Surrogates](https://mlsanity.com/paper/2305.19242/) (2023)
- [GP-BART: a novel Bayesian additive regression trees approach using Gaussian processes](https://mlsanity.com/paper/2204.02112/) (2022)
- [A spectral surrogate model for stochastic simulators computed from trajectory samples](https://mlsanity.com/paper/2207.05653/) (2022)
- [PCENet: High Dimensional Surrogate Modeling for Learning Uncertainty](https://mlsanity.com/paper/2202.05063/) (2022)
- [Model Mixing Using Bayesian Additive Regression Trees](https://mlsanity.com/paper/2301.02296/) (2023)
- [Co-Active Subspace Methods for the Joint Analysis of Adjacent Computer Models](https://mlsanity.com/paper/2311.18146/) (2023)
- [Discovering Active Subspaces for High-Dimensional Computer Models](https://mlsanity.com/paper/2307.11241/) (2023)
- [Analyzing Stochastic Computer Models: A Review with Opportunities](https://mlsanity.com/paper/2002.01321/) (2020)

## Built on by (most cited first, 3 of 3)

- [A Blueprint for Self-Evolving Coding Agents in Vehicle Aerodynamic Drag Prediction](https://mlsanity.com/paper/2603.21698/) (2026)
- [GNet: A scalable and flexible Gaussian process network with nonparametric neurons](https://mlsanity.com/paper/2607.10735/) (2026)
- [Adaptive multi-resolution Gaussian processes: Scalable exact inference with naturally data-sparse covariance matrices](https://mlsanity.com/paper/2609.30348/) (2026)

## Abstract (arXiv)

Accurate and efficient surrogate modeling is essential for modern computational science, and there are a staggering number of emulation methods to choose from. With new methods being developed all the time, comparing the relative strengths and weaknesses of different methods remains a challenge due to inconsistent benchmarking practices and (sometimes) limited reproducibility and transparency. In this work, we present a large-scale, fully reproducible comparison of 29 distinct emulators across 60 canonical test functions and 40 real emulation datasets. To facilitate rigorous, apples-to-apples comparisons, we introduce the R package textttduqling, which streamlines reproducible simulation studies using a consistent, simple syntax, and automatic internal scaling of inputs. This framework allows researchers to compare emulators in a unified environment and makes it possible to replicate or extend previous studies with minimal effort, even across different publications. Our results provide detailed empirical insight into the strengths and weaknesses of state-of-the-art emulators and offer guidance for both method developers and practitioners selecting a surrogate for new data. We discuss best practices for emulator comparison and highlight how textttduqling can accelerate research in emulator design and application.