# Principles and Guidelines for Randomized Controlled Trials in AI Evaluation

arXiv: https://arxiv.org/abs/2605.02050 · HTML: https://mlsanity.com/paper/2605.02050/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 40 references that are in the map.
- First posted on arXiv 3 May 2026.

## Builds on (most cited first, 10 of 10)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Toward an Evaluation Science for Generative AI Systems](https://mlsanity.com/paper/2503.05336/) (2025)
- [Evaluating General-Purpose AI with Psychometrics](https://mlsanity.com/paper/2310.16379/) (2023)
- [Limits of trust in medical AI](https://mlsanity.com/paper/2503.16692/) (2025)
- [Emerging Practices in Frontier AI Safety Frameworks](https://mlsanity.com/paper/2503.04746/) (2025)
- [Preliminary suggestions for rigorous GPAI model evaluations](https://mlsanity.com/paper/2508.00875/) (2025)
- [Measurement in AI Policy: Opportunities and Challenges](https://mlsanity.com/paper/2009.09071/) (2020)
- [Uplift modeling with continuous treatments: A predict-then-optimize approach](https://mlsanity.com/paper/2412.09232/) (2024)
- [Uplift vs. predictive modeling: a theoretical analysis](https://mlsanity.com/paper/2309.12036/) (2023)
- [Defining AI Models and AI Systems: A Framework to Resolve the Boundary Problem](https://mlsanity.com/paper/2603.10023/) (2026)

## Abstract (arXiv)

This work establishes a framework for standardizing AI evaluation RCTs (sometimes called human uplift studies). Drawing on established practices from disciplines with established RCT traditions, including software engineering, economics, clinical and health sciences, and psychology, we synthesize five principles drawn from established validity frameworks and open-science standards on transparency, repeatability, and verification, which together serve as the conceptual foundation for 33 actionable guidelines adapted for AI evaluation RCT contexts, expressed as requirements with rationales, implementation instructions, and evidence bases. We position the principles and guidelines as serving three key roles for AI evaluation RCTs: a design tool for planning studies, an evaluation rubric for assessing existing work, and a blueprint for standard setting as the field converges on norms. AI evaluation research currently lacks common standards and shared vocabulary for producing cumulative, comparable, policy-ready evidence. This framework is a contribution toward that foundation, providing evaluative criteria and a shared conceptual language alongside actionable guidelines.