# Online simulator-based experimental design for cognitive model selection

arXiv: https://arxiv.org/abs/2303.02227 · HTML: https://mlsanity.com/paper/2303.02227/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 88 references that are in the map.
- First posted on arXiv 3 Mar 2023.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows](https://mlsanity.com/paper/1805.07226/) (2018)
- [Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Design](https://mlsanity.com/paper/2103.02438/) (2021)
- [The Parallel Knowledge Gradient Method for Batch Bayesian Optimization](https://mlsanity.com/paper/1606.04414/) (2016)
- [Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models](https://mlsanity.com/paper/1501.03291/) (2015)
- [Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoods](https://mlsanity.com/paper/2111.02329/) (2021)
- [Bayesian Experimental Design for Implicit Models by Mutual Information Neural Estimation](https://mlsanity.com/paper/2002.08129/) (2020)
- [Optimizing Sequential Experimental Design with Deep Reinforcement Learning](https://mlsanity.com/paper/2202.00821/) (2022)
- [Gradient-based Bayesian Experimental Design for Implicit Models using Mutual Information Lower Bounds](https://mlsanity.com/paper/2105.04379/) (2021)
- [Unbiased and Efficient Log-Likelihood Estimation with Inverse Binomial Sampling](https://mlsanity.com/paper/2001.03985/) (2020)
- [Improving Artificial Teachers by Considering How People Learn and Forget](https://mlsanity.com/paper/2102.04174/) (2021)
- [Playtime Measurement with Survival Analysis](https://mlsanity.com/paper/1701.02359/) (2017)

## Built on by (most cited first, 3 of 3)

- [Closing the Loop to Discover Psychological Theories with an Automated Cognitive Scientist](https://mlsanity.com/paper/2606.26448/) (2026)
- [Simulation-Based Prior Knowledge Elicitation for Parametric Bayesian Models](https://mlsanity.com/paper/2308.11672/) (2023)
- [Accelerated Bayesian Optimal Experimental Design via Conditional Density Estimation and Informative Data](https://mlsanity.com/paper/2507.15235/) (2025)

## Abstract (arXiv)

The problem of model selection with a limited number of experimental trials has received considerable attention in cognitive science, where the role of experiments is to discriminate between theories expressed as computational models. Research on this subject has mostly been restricted to optimal experiment design with analytically tractable models. However, cognitive models of increasing complexity, with intractable likelihoods, are becoming more commonplace. In this paper, we propose BOSMOS: an approach to experimental design that can select between computational models without tractable likelihoods. It does so in a data-efficient manner, by sequentially and adaptively generating informative experiments. In contrast to previous approaches, we introduce a novel simulator-based utility objective for design selection, and a new approximation of the model likelihood for model selection. In simulated experiments, we demonstrate that the proposed BOSMOS technique can accurately select models in up to 2 orders of magnitude less time than existing LFI alternatives for three cognitive science tasks: memory retention, sequential signal detection and risky choice.