Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking

arXiv 2604.11581 ↗First posted 13 Apr 2026cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

LLM evaluations drive which models get deployed, what safety standards get adopted, which research conclusions get published, and how projections of AI's labor-market impact get made. Yet standard confidence intervals ignore variability from judge model choice, model temperature, and prompt phrasing, producing under-coverage that worsens with more data. The omitted variance can shift results enough to reverse conclusions citepbaumann2025llmhacking, huang2026dropping; pipelines that fail to average over it leave the surface that ``benchmark hacking'' exploits citepsingh2025leaderboard. This paper decomposes LLM pipeline uncertainty into its sources, distinguishes variance that shrinks with more data from sensitivity to researcher design choices, and uses design-study projections to reduce total evaluation error (TEE). Across the demonstrations, naive standard errors are 40 - 60% smaller than the TEE-corrected SE. Using Chatbot Arena data, we show naive 95% CI coverage drops as n grows while TEE-corrected coverage holds at 95%, and TEE-guided pipelines restrict the benchmark gaming surface from 56 to 32 Elo (K=27), below the human-leaderboard baseline. We show further that a small pilot recovers honest CIs and projects which design changes most improve precision. Acting on those projections halves MMLU estimation error against the answer key at equivalent cost, and raises per-match agreement with human votes by 7.9 percentage points on Chatbot Arena.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 42 of its 81 references that are in the map. First posted on arXiv 13 Apr 2026.

Builds on

42 of 81 references.

References · 42

2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2020Measuring Massive Multitask Language Understandingcited by 7,963
2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
2023Can Large Language Models Transform Computational Social Science?cited by 250
2024The Effect of Sampling Temperature on Problem Solving in Large Language Modelscited by 170

30 more

2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
2025Towards a Science of Scaling Agent Systemscited by 138
2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
2024Benchmarking Benchmark Leakage in Large Language Modelscited by 106
2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
2025The Leaderboard Illusioncited by 66
2025Foot-In-The-Door: A Multi-turn Jailbreak for LLMscited by 54
2025Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inferencecited by 53
2025Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotationcited by 50
2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 45
2025UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Makingcited by 42
2025Exploring the Impact of Temperature on Large Language Models:Hot or Cold?cited by 40
2023Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Modelscited by 39
2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
2025Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworkscited by 38
2025AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommonscited by 35
2026Brittlebench: Quantifying LLM robustness via prompt sensitivitycited by 21
2024Prompt Stability Scoring for Text Annotation with Large Language Modelscited by 15
2025Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarkscited by 10
2025Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankingscited by 8
2025Measuring all the noises of LLM Evalscited by 7
2025Quantitative LLM Judgescited by 6
2025Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial Attackscited by 6
2019TED-On: A Total Error Framework for Digital Traces of Human Behavior on Online Platformscited by 5
2025Exploring LLM Autoscoring Reliability in Large-Scale Writing Assessments Using Generalizability Theorycited by 4
2025Statistical Methods in Generative AIcited by 4
2025Risk Management for Mitigating Benchmark Failure Modes: BenchRiskcited by 3
2025Variance-Aware LLM Annotation for Strategy Research: Sources, Diagnostics, and a Protocol for Reliable Measurementcited by 3
2026Within-Model vs Between-Prompt Variability in Large Language Models for Creative Taskscited by 3
2026Revisiting Prompt Sensitivity in Large Language Models for Text Classification: The Role of Prompt Underspecificationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.