# Efficient Randomized Experiments Using Foundation Models

arXiv: https://arxiv.org/abs/2502.04262 · HTML: https://mlsanity.com/paper/2502.04262/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 58 references that are in the map.
- First posted on arXiv 6 Feb 2025.

## Builds on (most cited first, 23 of 23)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023)
- [PPI++: Efficient Prediction-Powered Inference](https://mlsanity.com/paper/2311.01453/) (2023)
- [Removing Hidden Confounding by Experimental Grounding](https://mlsanity.com/paper/1810.11646/) (2018)
- [Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data](https://mlsanity.com/paper/2410.13341/) (2024)
- [Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models](https://mlsanity.com/paper/2306.04746/) (2023)
- [Predictions as Surrogates: Revisiting Surrogate Outcomes in the Age of AI](https://mlsanity.com/paper/2501.09731/) (2025)
- [Combining Observational and Experimental Datasets Using Shrinkage Estimators](https://mlsanity.com/paper/2002.06708/) (2020)
- [Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation](https://mlsanity.com/paper/2406.04291/) (2024)
- [Adaptive Combination of Randomized and Observational Data](https://mlsanity.com/paper/2111.15012/) (2021)
- [End-To-End Causal Effect Estimation from Unstructured Natural Language Data](https://mlsanity.com/paper/2407.07018/) (2024)
- [Increasing the efficiency of randomized trial estimates via linear adjustment for a prognostic score](https://mlsanity.com/paper/2012.09935/) (2020)
- [Prediction-powered Generalization of Causal Inferences](https://mlsanity.com/paper/2406.02873/) (2024)
- [Falsification of Internal and External Validity in Observational Studies via Conditional Moment Restrictions](https://mlsanity.com/paper/2301.13133/) (2023)
- [An Omnibus Nonparametric Test of Equality in Distribution for Unknown Functions](https://mlsanity.com/paper/1510.04195/) (2015)
- [Hidden yet quantifiable: A lower bound for confounding strength using randomized trials](https://mlsanity.com/paper/2312.03871/) (2023)
- [TRIALSCOPE: A Unifying Causal Framework for Scaling Real-World Evidence Generation with Biomedical Language Models](https://mlsanity.com/paper/2311.01301/) (2023)
- [Detecting critical treatment effect bias in small subgroups](https://mlsanity.com/paper/2404.18905/) (2024)
- [Adaptive-TMLE for the Average Treatment Effect based on Randomized Controlled Trial Augmented with Real-World Data](https://mlsanity.com/paper/2405.07186/) (2024)
- [Prognostic Adjustment with Efficient Estimators to Unbiasedly Leverage Historical Data in Randomized Trials](https://mlsanity.com/paper/2305.19180/) (2023)
- [Data fusion for efficiency gain in ATE estimation: A practical review with simulations](https://mlsanity.com/paper/2407.01186/) (2024)
- [Robust integration of external control data in randomized trials](https://mlsanity.com/paper/2406.17971/) (2024)

## Built on by (most cited first, 10 of 10)

- [Prediction-Powered Causal Inferences](https://mlsanity.com/paper/2502.06343/) (2025)
- [General Synthetic-Powered Inference](https://mlsanity.com/paper/2509.20345/) (2025)
- [How can we assess human-agent interactions? Case studies in software agent design](https://mlsanity.com/paper/2510.09801/) (2025)
- [Revisiting Active Sequential Prediction-Powered Mean Estimation](https://mlsanity.com/paper/2604.18569/) (2026)
- [Epidemiology of Large Language Models: A Benchmark for Observational Distribution Knowledge](https://mlsanity.com/paper/2511.03070/) (2025)
- [Generative Synthetic Data for Causal Inference: Pitfalls, Remedies, and Opportunities](https://mlsanity.com/paper/2604.23904/) (2026)
- [Robust estimation of heterogeneous treatment effects in randomized trials leveraging external data](https://mlsanity.com/paper/2507.03681/) (2025)
- [SADA: Safe and Adaptive Aggregation of Multiple Black-Box Predictions in Semi-Supervised Learning](https://mlsanity.com/paper/2509.21707/) (2025)
- [A Technical Exploration of Causal Inference with Hybrid LLM Synthetic Data](https://mlsanity.com/paper/2511.00318/) (2025)
- [Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation](https://mlsanity.com/paper/2605.31278/) (2026)

## Abstract (arXiv)

Randomized experiments are the preferred approach for evaluating the effects of interventions, but they are costly and often yield estimates with substantial uncertainty. On the other hand, in silico experiments leveraging foundation models offer a cost-effective alternative that can potentially attain higher statistical precision. However, the benefits of in silico experiments come with a significant risk: statistical inferences are not valid if the models fail to accurately predict experimental responses to interventions. In this paper, we propose a novel approach that integrates the predictions from multiple foundation models with experimental data while preserving valid statistical inference. Our estimator is consistent and asymptotically normal, with asymptotic variance no larger than the standard estimator based on experimental data alone. Importantly, these statistical properties hold even when model predictions are arbitrarily biased. Empirical results across several randomized experiments show that our estimator offers substantial precision gains, equivalent to a reduction of up to 20% in the sample size needed to match the same precision as the standard estimator based on experimental data alone.