# Preliminary suggestions for rigorous GPAI model evaluations

arXiv: https://arxiv.org/abs/2508.00875 · HTML: https://mlsanity.com/paper/2508.00875/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 17 references that are in the map.
- First posted on arXiv 22 Jul 2025.

## Builds on (most cited first, 5 of 5)

- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023)
- [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024)
- [Targeting the Benchmark: On Methodology in Current Natural Language Processing Research](https://mlsanity.com/paper/2007.04792/) (2020)
- [The State of Applied Econometrics - Causality and Policy Evaluation](https://mlsanity.com/paper/1607.00699/) (2016)
- [What Makes an Evaluation Useful? Common Pitfalls and Best Practices](https://mlsanity.com/paper/2503.23424/) (2025)

## Built on by (most cited first, 5 of 5)

- [STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reports](https://mlsanity.com/paper/2508.09853/) (2025)
- [Against racing to AGI: Cooperation, deterrence, and catastrophic risks](https://mlsanity.com/paper/2507.21839/) (2025)
- [Measuring Biological Capabilities and Risks of AI Agents](https://mlsanity.com/paper/2606.19899/) (2026)
- [RCTs for Frontier AI Governance: Methodological Challenges and Solutions for Human Uplift Studies](https://mlsanity.com/paper/2603.11001/) (2026)
- [Principles and Guidelines for Randomized Controlled Trials in AI Evaluation](https://mlsanity.com/paper/2605.02050/) (2026)

## Abstract (arXiv)

This document presents a preliminary compilation of general-purpose AI (GPAI) evaluation practices that may promote internal validity, external validity and reproducibility. It includes suggestions for human uplift studies and benchmark evaluations, as well as cross-cutting suggestions that may apply to many different evaluation types. Suggestions are organised across four stages in the evaluation life cycle: design, implementation, execution and documentation. Drawing from established practices in machine learning, statistics, psychology, economics, biology and other fields recognised to have important lessons for AI evaluation, these suggestions seek to contribute to the conversation on the nascent and evolving field of the science of GPAI evaluations. The intended audience of this document includes providers of GPAI models presenting systemic risk (GPAISR), for whom the EU AI Act lays out specific evaluation requirements; third-party evaluators; policymakers assessing the rigour of evaluations; and academic researchers developing or conducting GPAI evaluations.