# BayesAME: Bayesian Active Model Evaluation

arXiv: https://arxiv.org/abs/2607.27023 · HTML: https://mlsanity.com/paper/2607.27023/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 29 Jul 2026.

## Builds on (most cited first, 24 of 24)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning](https://mlsanity.com/paper/2310.16049/) (2023)
- [Anchor Points: Benchmarking Models with Much Fewer Examples](https://mlsanity.com/paper/2309.08638/) (2023)
- [Active Testing: Sample-Efficient Model Evaluation](https://mlsanity.com/paper/2103.05331/) (2021)
- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models](https://mlsanity.com/paper/2407.12844/) (2024)
- [How Benchmark Prediction from Fewer Data Misses the Mark](https://mlsanity.com/paper/2506.07673/) (2025)
- [β3-IRT: A New Item Response Model and its Applications](https://mlsanity.com/paper/1903.04016/) (2019)
- [Scaling Up Active Testing to Large Language Models](https://mlsanity.com/paper/2508.09093/) (2025)
- [Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMs](https://mlsanity.com/paper/2505.03814/) (2025)
- [Scales++: Compute Efficient Evaluation Subset Selection with Cognitive Scales Embeddings](https://mlsanity.com/paper/2510.26384/) (2025)
- [Toward a unified framework for data-efficient evaluation of large language models](https://mlsanity.com/paper/2510.04051/) (2025)
- [Active Testing of Large Language Models via Approximate Neyman Allocation](https://mlsanity.com/paper/2605.10075/) (2026)
- [Can We Trust Item Response Theory for AI Evaluation?](https://mlsanity.com/paper/2607.15190/) (2026)
- [Efficient Sequential Evaluation of Large Language Models](https://mlsanity.com/paper/2607.17409/) (2026)
- [CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion](https://mlsanity.com/paper/2607.05046/) (2026)
- [ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation](https://mlsanity.com/paper/2604.23099/) (2026)
- [Efficient Benchmarking Is Just Feature Selection and Multiple Regression](https://mlsanity.com/paper/2605.25773/) (2026)

## Abstract (arXiv)

Evaluating large generative models across benchmarks is time-consuming and computationally expensive. This drives the need for methods that can estimate full benchmark performance by evaluating models on only a subset of items, known as a coreset. Current literature mostly requires the practitioner to input a coreset size. However, when reliable performance estimation takes priority over efficiency, an evaluation method should also be capable of automatically determining a coreset size that reflects this priority. We introduce BayesAME, a sequential Bayesian framework specifically targeting automatic determination of the coreset size. BayesAME models performance as a random variable by defining a latent ability for each group of items sharing the same historical model performances, with a joint prior distribution encoding the belief that the target model behaves similarly to these historical models. The posterior distribution over these abilities is used to derive performance estimators, quantify performance uncertainty, and select items to add to the coreset via an information-gain criterion. The coreset is iteratively augmented until the performance estimate fluctuation and the performance uncertainty fall below their respective user-defined thresholds. We propose a multi-target extension that captures performance correlations across multiple target models to further reduce the coreset size. Through extensive experiments across diverse benchmarks, we demonstrate that BayesAME consistently outperforms sequential adaptations of existing methods. Crucially, our comprehensive analysis addresses recent skepticism in the literature, establishing that non-random coreset selection is advantageous over random selection. Finally, we highlight that leveraging continuous response log-likelihoods over traditional binary scores significantly enhances estimation accuracy.