# Benchmarking Simulation-Based Inference

arXiv: https://arxiv.org/abs/2101.04653 · HTML: https://mlsanity.com/paper/2101.04653/

## Key facts

- Cited by 162 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 123 references that are in the map.
- First posted on arXiv 12 Jan 2021.

## Builds on (most cited first, 30 of 55)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Revisiting Classifier Two-Sample Tests](https://mlsanity.com/paper/1610.06545/) (2016)
- [A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluation](https://mlsanity.com/paper/1602.03253/) (2016)
- [Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows](https://mlsanity.com/paper/1805.07226/) (2018)
- [Automatic Posterior Transformation for Likelihood-Free Inference](https://mlsanity.com/paper/1905.07488/) (2019)
- [Flexible statistical inference for mechanistic models of neural dynamics](https://mlsanity.com/paper/1711.01861/) (2017)
- [BayesFlow: Learning complex stochastic models with invertible neural networks](https://mlsanity.com/paper/2003.06281/) (2020)
- [A Kernel Test of Goodness of Fit](https://mlsanity.com/paper/1602.02964/) (2016)
- [Learning Deep Kernels for Non-Parametric Two-Sample Tests](https://mlsanity.com/paper/2002.09116/) (2020)
- [Validating Bayesian Inference Algorithms with Simulation-Based Calibration](https://mlsanity.com/paper/1804.06788/) (2018)
- [Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy](https://mlsanity.com/paper/1611.04488/) (2016)
- [On Contrastive Learning for Likelihood-free Inference](https://mlsanity.com/paper/2002.03712/) (2020)
- [Approximating Likelihood Ratios with Calibrated Discriminative Classifiers](https://mlsanity.com/paper/1506.02169/) (2015)
- [Likelihood-free inference by ratio estimation](https://mlsanity.com/paper/1611.10242/) (2016)
- [Neural Approximate Sufficient Statistics for Implicit Models](https://mlsanity.com/paper/2010.10079/) (2020)
- [A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks](https://mlsanity.com/paper/1912.10481/) (2019)
- [Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models](https://mlsanity.com/paper/1501.03291/) (2015)
- [Likelihood-free inference with emulator networks](https://mlsanity.com/paper/1805.09294/) (2018)
- [Mining gold from implicit models to improve likelihood-free inference](https://mlsanity.com/paper/1805.12244/) (2018)
- [Fast ε-free Inference of Simulation Models with Bayesian Conditional Density Estimation](https://mlsanity.com/paper/1605.06376/) (2016)

## Built on by (most cited first, 30 of 162)

- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Flow Matching for Scalable Simulation-Based Inference](https://mlsanity.com/paper/2305.17161/) (2023)
- [All-in-one simulation-based inference](https://mlsanity.com/paper/2404.09636/) (2024)
- [Sequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Models](https://mlsanity.com/paper/2210.04872/) (2022)
- [Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks](https://mlsanity.com/paper/2112.08866/) (2021)
- [Compositional Score Modeling for Simulation-based Inference](https://mlsanity.com/paper/2209.14249/) (2022)
- [Robust Neural Posterior Estimation and Statistical Model Criticism](https://mlsanity.com/paper/2210.06564/) (2022)
- [Sampling-Based Accuracy Testing of Posterior Estimators for General Inference](https://mlsanity.com/paper/2302.03026/) (2023)
- [Investigating the Impact of Model Misspecification in Neural Simulation-based Inference](https://mlsanity.com/paper/2209.01845/) (2022)
- [Variational methods for simulation-based inference](https://mlsanity.com/paper/2203.04176/) (2022)
- [Truncated proposals for scalable and hassle-free simulation-based inference](https://mlsanity.com/paper/2210.04815/) (2022)
- [JANA: Jointly Amortized Neural Approximation of Complex Bayesian Models](https://mlsanity.com/paper/2302.09125/) (2023)
- [Neural Methods for Amortized Inference](https://mlsanity.com/paper/2404.12484/) (2024)
- [GATSBI: Generative Adversarial Training for Simulation-Based Inference](https://mlsanity.com/paper/2203.06481/) (2022)
- [Simulation-Based Inference: A Practical Guide](https://mlsanity.com/paper/2508.12939/) (2025)
- [Sequential Neural Posterior and Likelihood Approximation](https://mlsanity.com/paper/2102.06522/) (2021)
- [Truncated Marginal Neural Ratio Estimation](https://mlsanity.com/paper/2107.01214/) (2021)
- [Neural Networks for Parameter Estimation in Intractable Models](https://mlsanity.com/paper/2107.14346/) (2021)
- [A Trust Crisis In Simulation-Based Inference? Your Posterior Approximations Can Be Unfaithful](https://mlsanity.com/paper/2110.06581/) (2021)
- [Towards Reliable Simulation-Based Inference with Balanced Neural Ratio Estimation](https://mlsanity.com/paper/2208.13624/) (2022)
- [Benchopt: Reproducible, efficient and collaborative optimization benchmarks](https://mlsanity.com/paper/2206.13424/) (2022)
- [Addressing Misspecification in Simulation-based Inference through Data-driven Calibration](https://mlsanity.com/paper/2405.08719/) (2024)
- [Free-form Flows: Make Any Architecture a Normalizing Flow](https://mlsanity.com/paper/2310.16624/) (2023)
- [Conformal Contextual Robust Optimization](https://mlsanity.com/paper/2310.10003/) (2023)
- [Amortized Probabilistic Conditioning for Optimization, Simulation and Inference](https://mlsanity.com/paper/2410.15320/) (2024)
- [Consistency Models for Scalable and Fast Simulation-Based Inference](https://mlsanity.com/paper/2312.05440/) (2023)
- [Diffusion posterior sampling for simulation-based inference in tall data settings](https://mlsanity.com/paper/2404.07593/) (2024)
- [L-C2ST: Local Diagnostics for Posterior Approximations in Simulation-Based Inference](https://mlsanity.com/paper/2306.03580/) (2023)
- [Adversarial robustness of amortized Bayesian inference](https://mlsanity.com/paper/2305.14984/) (2023)
- [Generalized Bayesian Inference for Scientific Simulators via Amortized Cost Estimation](https://mlsanity.com/paper/2305.15208/) (2023)

## Abstract (arXiv)

Recent advances in probabilistic modelling have led to a large number of simulation-based inference algorithms which do not require numerical evaluation of likelihoods. However, a public benchmark with appropriate performance metrics for such 'likelihood-free' algorithms has been lacking. This has made it difficult to compare algorithms and identify their strengths and weaknesses. We set out to fill this gap: We provide a benchmark with inference tasks and suitable performance metrics, with an initial selection of algorithms including recent approaches employing neural networks and classical Approximate Bayesian Computation methods. We found that the choice of performance metric is critical, that even state-of-the-art algorithms have substantial room for improvement, and that sequential estimation improves sample efficiency. Neural network-based approaches generally exhibit better performance, but there is no uniformly best algorithm. We provide practical advice and highlight the potential of the benchmark to diagnose problems and improve algorithms. The results can be explored interactively on a companion website. All code is open source, making it possible to contribute further benchmark tasks and inference algorithms.