mlsanityData to 7 Oct 2026

Benchmarking Simulation-Based Inference

arXiv 2101.04653 ↗First posted 12 Jan 2021stat.ML cs.LGcited by 160

Not on the map. Why?

Abstract (arXiv)

Recent advances in probabilistic modelling have led to a large number of simulation-based inference algorithms which do not require numerical evaluation of likelihoods. However, a public benchmark with appropriate performance metrics for such 'likelihood-free' algorithms has been lacking. This has made it difficult to compare algorithms and identify their strengths and weaknesses. We set out to fill this gap: We provide a benchmark with inference tasks and suitable performance metrics, with an initial selection of algorithms including recent approaches employing neural networks and classical Approximate Bayesian Computation methods. We found that the choice of performance metric is critical, that even state-of-the-art algorithms have substantial room for improvement, and that sequential estimation improves sample efficiency. Neural network-based approaches generally exhibit better performance, but there is no uniformly best algorithm. We provide practical advice and highlight the potential of the benchmark to diagnose problems and improve algorithms. The results can be explored interactively on a companion website. All code is open source, making it possible to contribute further benchmark tasks and inference algorithms.

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). Builds on 55 of its 123 references that are in the map. First posted on arXiv 12 Jan 2021.

Builds on

55 of 123 references.

References · 55

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2015Variational Inference with Normalizing Flowscited by 2,161
  6. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  7. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  8. 2019Neural Spline Flowscited by 590
  9. 2019The frontier of simulation-based inferencecited by 490
  10. 2018Pyro: Deep Universal Probabilistic Programmingcited by 379
  11. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  12. 2016Revisiting Classifier Two-Sample Testscited by 258
38 more
  1. 2016A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluationcited by 236
  2. 2018Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flowscited by 217
  3. 2019Automatic Posterior Transformation for Likelihood-Free Inferencecited by 216
  4. 2017Flexible statistical inference for mechanistic models of neural dynamicscited by 162
  5. 2016A Kernel Test of Goodness of Fitcited by 143
  6. 2020BayesFlow: Learning complex stochastic models with invertible neural networkscited by 139
  7. 2020Learning Deep Kernels for Non-Parametric Two-Sample Testscited by 129
  8. 2018Validating Bayesian Inference Algorithms with Simulation-Based Calibrationcited by 118
  9. 2016Generative Models and Model Criticism via Optimized Maximum Mean Discrepancycited by 99
  10. 2020On Contrastive Learning for Likelihood-free Inferencecited by 96
  11. 2015Approximating Likelihood Ratios with Calibrated Discriminative Classifierscited by 87
  12. 2016Likelihood-free inference by ratio estimationcited by 80
  13. 2020Neural Approximate Sufficient Statistics for Implicit Modelscited by 72
  14. 2019A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Taskscited by 62
  15. 2015Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Modelscited by 61
  16. 2018Likelihood-free inference with emulator networkscited by 61
  17. 2018Mining gold from implicit models to improve likelihood-free inferencecited by 60
  18. 2016Fast ε-free Inference of Simulation Models with Bayesian Conditional Density Estimationcited by 50
  19. 2017A Linear-Time Kernel Goodness-of-Fit Testcited by 41
  20. 2016Inference Compilation and Universal Probabilistic Programmingcited by 31
  21. 2018A Likelihood-Free Inference Framework for Population Genetic Data using Exchangeable Neural Networkscited by 30
  22. 2018Constraining Effective Field Theories with Machine Learningcited by 26
  23. 2014High-Dimensional Density Ratio Estimation with Extensions to Approximate Likelihood Computationcited by 19
  24. 2019Parallel Gaussian process surrogate Bayesian inference with noisy likelihood evaluationscited by 18
  25. 2019Etalumis: Bringing Probabilistic Programming to Scientific Simulators at Scalecited by 18
  26. 2014GPS-ABC: Gaussian Process Surrogate Approximate Bayesian Computationcited by 17
  27. 2017Efficient acquisition rules for model-based approximate Bayesian computationcited by 16
  28. 2014Likelihood-free inference via classificationcited by 13
  29. 2016ABC random forests for Bayesian parameter inferencecited by 11
  30. 2018Sequential Neural Methods for Likelihood-free Inferencecited by 11
  31. 2013Diagnostic tools of approximate Bayesian computation using the coverage propertycited by 10
  32. 2017ELFI: Engine for Likelihood-Free Inferencecited by 10
  33. 2019Likelihood-free MCMC with Amortized Approximate Ratio Estimatorscited by 9
  34. 2019Adaptive Approximate Bayesian Computation Tolerance Selectioncited by 8
  35. 2014Accelerating ABC methods using Gaussian processescited by 7
  36. 2018Dynamic Likelihood-free Inference via Ratio Estimation (DIRE)cited by 7
  37. 2019Likelihood-free approximate Gibbs samplingcited by 7
  38. 2018Approximating the Likelihood in Approximate Bayesian Computationcited by 5

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 22; 2023: 30; 2024: 32; 2025: 29; 2026: 35.

160 papers.

Sort:

By area:

  1. 2021Transformers Can Do Bayesian Inferencecited by 338
  2. 2023Flow Matching for Scalable Simulation-Based Inferencecited by 97
  3. 2024All-in-one simulation-based inferencecited by 71
  4. 2022Sequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Modelscited by 50
  5. 2021Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networkscited by 48
  6. 2022Compositional Score Modeling for Simulation-based Inferencecited by 46
  7. 2022Robust Neural Posterior Estimation and Statistical Model Criticismcited by 45
  8. 2023Sampling-Based Accuracy Testing of Posterior Estimators for General Inferencecited by 45
  9. 2022Investigating the Impact of Model Misspecification in Neural Simulation-based Inferencecited by 44
  10. 2022Variational methods for simulation-based inferencecited by 43
  11. 2023JANA: Jointly Amortized Neural Approximation of Complex Bayesian Modelscited by 41
  12. 2022Truncated proposals for scalable and hassle-free simulation-based inferencecited by 40
38 more
  1. 2022GATSBI: Generative Adversarial Training for Simulation-Based Inferencecited by 36
  2. 2024Neural Methods for Amortized Inferencecited by 36
  3. 2021Sequential Neural Posterior and Likelihood Approximationcited by 30
  4. 2021Truncated Marginal Neural Ratio Estimationcited by 30
  5. 2025Simulation-Based Inference: A Practical Guidecited by 30
  6. 2021Neural Networks for Parameter Estimation in Intractable Modelscited by 28
  7. 2021A Trust Crisis In Simulation-Based Inference? Your Posterior Approximations Can Be Unfaithfulcited by 27
  8. 2022Benchopt: Reproducible, efficient and collaborative optimization benchmarkscited by 27
  9. 2022Towards Reliable Simulation-Based Inference with Balanced Neural Ratio Estimationcited by 27
  10. 2023Free-form Flows: Make Any Architecture a Normalizing Flowcited by 26
  11. 2024Addressing Misspecification in Simulation-based Inference through Data-driven Calibrationcited by 26
  12. 2023Conformal Contextual Robust Optimizationcited by 25
  13. 2023Consistency Models for Scalable and Fast Simulation-Based Inferencecited by 22
  14. 2024Amortized Probabilistic Conditioning for Optimization, Simulation and Inferencecited by 22
  15. 2023Adversarial robustness of amortized Bayesian inferencecited by 17
  16. 2023Generalized Bayesian Inference for Scientific Simulators via Amortized Cost Estimationcited by 17
  17. 2023L-C2ST: Local Diagnostics for Posterior Approximations in Simulation-Based Inferencecited by 17
  18. 2024Diffusion posterior sampling for simulation-based inference in tall data settingscited by 17
  19. 2021Group equivariant neural posterior estimationcited by 16
  20. 2022Simulator-Based Inference with Waldo: Confidence Regions by Leveraging Prediction Algorithms and Posterior Estimators for Inverse Problemscited by 14
  21. 2023Leveraging Self-Consistency for Data-Efficient Amortized Bayesian Inferencecited by 14
  22. 2024A Practical Guide to Sample-based Statistical Distances for Evaluating Generative Models in Sciencecited by 14
  23. 2021Fast and Credible Likelihood-Free Cosmology with Truncated Marginal Neural Ratio Estimationcited by 13
  24. 2022Black-box Bayesian inference for economic agent-based modelscited by 13
  25. 2023Optimally-Weighted Estimators of the Maximum Mean Discrepancy for Likelihood-Free Inferencecited by 13
  26. 2025Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Modelscited by 13
  27. 2024Amortized Bayesian Multilevel Modelscited by 12
  28. 2023Simulation-based Inference for Cardiovascular Modelscited by 11
  29. 2022E-Valuating Classifier Two-Sample Testscited by 10
  30. 2023Calibrating Neural Simulation-Based Inference with Differentiable Coverage Probabilitycited by 10
  31. 2022Neural Posterior Estimation with Differentiable Simulatorscited by 9
  32. 2024Sourcerer: Sample-based Maximum Entropy Source Distribution Estimationcited by 9
  33. 2025Multifidelity Simulation-based Inference for Computationally Expensive Simulatorscited by 9
  34. 2025Simulation-based Bayesian inference under model misspecificationcited by 9
  35. 2022Amortised Likelihood-free Inference for Expensive Time-series Simulators with Signatured Ratio Estimationcited by 8
  36. 2022Validation Diagnostics for SBI algorithms based on Normalizing Flowscited by 8
  37. 2023Misspecification-robust Sequential Neural Likelihood for Simulation-based Inferencecited by 8
  38. 2023Simulation-based Inference for High-dimensional Data using Surjective Sequential Neural Likelihood Estimationcited by 8

50 most cited of 160 All 160 (JSON)