# How Much Can Repeated Running Buy? A Measured Variance Ceiling for LLM Code Evaluation

arXiv: https://arxiv.org/abs/2606.00920 · HTML: https://mlsanity.com/paper/2606.00920/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 39 references that are in the map.
- First posted on arXiv 30 May 2026.

## Builds on (most cited first, 13 of 13)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation](https://mlsanity.com/paper/2308.15363/) (2023)
- [Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs](https://mlsanity.com/paper/2305.11860/) (2023)
- [Non-Determinism of "Deterministic" LLM Settings](https://mlsanity.com/paper/2408.04667/) (2024)
- [Do Repetitions Matter? Strengthening Reliability in LLM Evaluations](https://mlsanity.com/paper/2509.24086/) (2025)

## Built on by (most cited first, 5 of 5)

- [Cross-Model LLM Code Review: Should you use Claude to review Codex or vice versa?](https://mlsanity.com/paper/2607.21656/) (2026)
- [Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity](https://mlsanity.com/paper/2608.02665/) (2026)
- [The History Is the Detector: Executing CVE Patch History, End-to-End](https://mlsanity.com/paper/2609.05335/) (2026)
- [Style Over Substance: Content-Invariant Wrappers Flip LLM Safety-Judge Verdicts](https://mlsanity.com/paper/2609.08236/) (2026)
- [A Trust Layer for Agent Evaluation](https://mlsanity.com/paper/2610.07274/) (2026)

## Abstract (arXiv)

A benchmark score is a measurement whose uncertainty has two sources: an aleatoric component from stochastic decoding and an epistemic component from the finite problem set. Using 16 models on 99 recent LeetCode-style problems with two prompts and five runs each (15,840 judged runs in a fully crossed design), we estimate both components and ask a design question: how much can repeated running buy? First, the pooled-binomial interval that repeated-run protocols invite answers neither question of interest---it is about 2× too narrow for generalisation to new problems and 2× too wide for the fixed problem set---and under correctly clustered intervals, 12 of 15 adjacent leaderboard pairs are statistically unresolved. Second, the variance ceiling 1+μw/σb2, the largest factor by which repetition can reduce score variance, has median 1.34 across models (range 1.05--1.71), so repetition can narrow confidence intervals by at most 23% on this benchmark, and by 14% for the median model. Third, once the cost of adding verified problems is priced in, the precision-optimal repeat count is R*=√μw cp/(σb2 cg), and the variance curve is flat around it: on this benchmark the decisive reason to repeat is not precision but identification---two to three runs suffice to estimate the components that the ceiling and the fixed-benchmark interval require. All three are measurements of one benchmark at one decoding temperature, and we state explicitly what they do and do not license.