# CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion

arXiv: https://arxiv.org/abs/2607.05046 · HTML: https://mlsanity.com/paper/2607.05046/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 33 references that are in the map.
- First posted on arXiv 6 Jul 2026.

## Builds on (most cited first, 9 of 9)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [PPI++: Efficient Prediction-Powered Inference](https://mlsanity.com/paper/2311.01453/) (2023)
- [LLM360: Towards Fully Transparent Open-Source LLMs](https://mlsanity.com/paper/2312.06550/) (2023)
- [AutoEval Done Right: Using Synthetic Data for Model Evaluation](https://mlsanity.com/paper/2403.07008/) (2024)
- [Can Unconfident LLM Annotations Be Used for Confident Conclusions?](https://mlsanity.com/paper/2408.15204/) (2024)
- [Cost-Optimal Active AI Model Evaluation](https://mlsanity.com/paper/2506.07949/) (2025)
- [LLM360 K2: Building a 65B 360-Open-Source Large Language Model from Scratch](https://mlsanity.com/paper/2501.07124/) (2025)
- [Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization](https://mlsanity.com/paper/2603.02029/) (2026)

## Built on by (most cited first, 3 of 3)

- [Efficient Sequential Evaluation of Large Language Models](https://mlsanity.com/paper/2607.17409/) (2026)
- [BayesAME: Bayesian Active Model Evaluation](https://mlsanity.com/paper/2607.27023/) (2026)
- [Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models](https://mlsanity.com/paper/2610.18248/) (2026)

## Abstract (arXiv)

Evaluating generative AI models is a routine, but resource-intensive, process that is conducted over and over again during the course of model development. In this work, we propose Collaborative Evaluation (CollabEval), a simple, effective, and principled method for exploiting dependencies between historical runs of different models on the same tasks to improve statistical efficiency. Specifically, our approach treats model evaluation as a matrix completion problem over an M × N matrix of evaluation scores, where M is the total number of models and N is the total number of evaluation prompts. We assume that a subset of these M models are targeted for evaluation. For these target models only a small fraction, p, of prompts has been annotated with evaluation scores. Leveraging recent results in prediction-powered inference, we build a low-rank approximation of the score matrix, and use the reconstructed values as control variates in a manner that guarantees unbiased estimates of the true evaluation metric mean, in addition to statistically valid confidence intervals. Empirically, across a wide range of datasets, models, and sparsity levels p, we find that CollabEval substantially reduces the mean confidence interval size, and the mean squared error of the point estimate, compared to baseline methods at the same annotation budget.