# Using simulation studies to evaluate statistical methods

arXiv: https://arxiv.org/abs/1712.03198 · HTML: https://mlsanity.com/paper/1712.03198/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Dec 2017.

## Built on by (most cited first, 30 of 49)

- [Shrinkage Bayesian Causal Forests for Heterogeneous Treatment Effects Estimation](https://mlsanity.com/paper/2102.06573/) (2021)
- [Linguistically inspired roadmap for building biologically reliable protein language models](https://mlsanity.com/paper/2207.00982/) (2022)
- [On the role of benchmarking data sets and simulations in method comparison studies](https://mlsanity.com/paper/2208.01457/) (2022)
- [Thematic Analysis with Open-Source Generative AI and Machine Learning: A New Method for Inductive Qualitative Codebook Development](https://mlsanity.com/paper/2410.03721/) (2024)
- [Can We Trust Item Response Theory for AI Evaluation?](https://mlsanity.com/paper/2607.15190/) (2026)
- [A Fast Bootstrap Algorithm for Causal Inference with Large Data](https://mlsanity.com/paper/2302.02859/) (2023)
- [SpaCE: The Spatial Confounding Environment](https://mlsanity.com/paper/2312.00710/) (2023)
- [Deep Nonparametric Conditional Independence Tests for Images](https://mlsanity.com/paper/2411.06140/) (2024)
- [Generalized Prediction-Powered Inference, with Application to Binary Classifier Evaluation](https://mlsanity.com/paper/2602.10332/) (2026)
- [Opening the random forest black box by the analysis of the mutual impact of features](https://mlsanity.com/paper/2304.02490/) (2023)
- [An engine to simulate insurance fraud network data](https://mlsanity.com/paper/2308.11659/) (2023)
- [Adapting tree-based multiple imputation methods for multi-level data? A simulation study](https://mlsanity.com/paper/2401.14161/) (2024)
- [A flexible Bayesian g-formula for causal survival analyses with time-dependent confounding](https://mlsanity.com/paper/2402.02306/) (2024)
- [Understanding overfitting in random forest for probability estimation: a visualization and simulation study](https://mlsanity.com/paper/2402.18612/) (2024)
- [Performance of Cross-Validated Targeted Maximum Likelihood Estimation](https://mlsanity.com/paper/2409.11265/) (2024)
- [Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replication](https://mlsanity.com/paper/2501.06826/) (2025)
- [Comparison of the Cox proportional hazards model and Random Survival Forest algorithm for predicting patient-specific survival probabilities in clinical trial data](https://mlsanity.com/paper/2502.03119/) (2025)
- [Gradient Boosting for Spatial Regression Models with Autoregressive Disturbances](https://mlsanity.com/paper/2506.13682/) (2025)
- [Analyzing Error Sources in Global Feature Effect Estimation](https://mlsanity.com/paper/2603.15057/) (2026)
- [The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains](https://mlsanity.com/paper/2605.11205/) (2026)
- [In-Context Learning for the Imputation of Public Opinion Data with Large Language Models](https://mlsanity.com/paper/2606.09351/) (2026)
- [Embedded Conditional Independence Tests for Large Language Model Generated Text with an Application to German Parliament Speeches](https://mlsanity.com/paper/2609.00946/) (2026)
- [Three Routes to One Answer: Reconciling AIPW, TMLE, and Double Machine Learning for Applied Researchers](https://mlsanity.com/paper/2609.26142/) (2026)
- [Leveraging Historical Data for High-Dimensional Regression Adjustment, a Composite Covariate Approach](https://mlsanity.com/paper/2103.14421/) (2021)
- [Hybrid Feature- and Similarity-Based Models for Joint Prediction and Interpretation](https://mlsanity.com/paper/2204.06076/) (2022)
- [DagSim: Combining DAG-based model structure with unconstrained data types and relations for flexible, transparent, and modularized data simulation](https://mlsanity.com/paper/2205.11234/) (2022)
- [Evaluation of network-guided random forest for disease gene discovery](https://mlsanity.com/paper/2308.01323/) (2023)
- [Effect of hyperparameters on variable selection in random forests](https://mlsanity.com/paper/2309.06943/) (2023)
- [Predictive Analytics of Varieties of Potatoes](https://mlsanity.com/paper/2404.03701/) (2024)
- [High-dimensional multiple imputation (HDMI) for partially observed confounders including natural language processing-derived auxiliary covariates](https://mlsanity.com/paper/2405.10925/) (2024)

## Abstract (arXiv)

Simulation studies are computer experiments that involve creating data by pseudorandom sampling. The key strength of simulation studies is the ability to understand the behaviour of statistical methods because some 'truth' (usually some parameter/s of interest) is known from the process of generating the data. This allows us to consider properties of methods, such as bias. While widely used, simulation studies are often poorly designed, analysed and reported. This tutorial outlines the rationale for using simulation studies and offers guidance for design, execution, analysis, reporting and presentation. In particular, this tutorial provides: a structured approach for planning and reporting simulation studies, which involves defining aims, data-generating mechanisms, estimands, methods and performance measures ('ADEMP'); coherent terminology for simulation studies; guidance on coding simulation studies; a critical discussion of key performance measures and their estimation; guidance on structuring tabular and graphical presentation of results; and new graphical presentations. With a view to describing recent practice, we review 100 articles taken from Volume 34 of Statistics in Medicine that included at least one simulation study and identify areas for improvement.