When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits
Not on the map. Why?
Abstract (arXiv)
Behavioral contamination detectors can return "no evidence" either because a benchmark is clean or because the audit has little power. We formalize this distinction for a benchmark in which an unknown fraction alpha of items was seen during training. With matched clean and seen controls, the behavioral channel is the sparse mixture Q_alpha = (1 - alpha) P_0 + alpha P_1, and an exact second-moment argument shows that detectability is governed by alpha * rho * sqrt(m), where rho^2 = chi^2(P_1 || P_0) measures behavioral separability. Any scalar detector reduces to its efficacy, ef = |E_1 f - E_0 f| / sqrt(Var_0(f)) <= rho, which can be estimated from controls before the audit is run. A separate sample-split certificate lower-bounds alpha distribution-free, without requiring an orientation assumption. Our empirical finding is two-sided. Frozen calibration efficacy predicts held-out power curves, with R^2 = 0.83-0.98 across six exact-permutation channels, but the efficacy-only Gaussian budget is miscalibrated at the small sample sizes it prescribes, failing in 9/9 gate-passing channels even though efficacy itself transports. The failure is in the inversion, not the calibration. A predeclared two-stage planner that simulates the deployed test repairs the budgets, is uniformly conservative, and abstains when its probe does not transport. The certificate is valid but vacuous at audit scale, and a five-seed paired injection study recovers the mechanism ordering verbatim > paraphrase > surface, in which the apparent answer-only signal is explained by baseline drift. We report the audit contract and its failures together: a non-rejection is interpretable only alongside the efficacy, budget, and validity gates that produced it.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 38 references that are in the map. First posted on arXiv 8 Aug 2026.
Builds on
17 of 38 references.
References · 17
- 2020Extracting Training Data from Large Language Modelscited by 2,189
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
- 2024OLMo: Accelerating the Science of Language Modelscited by 613
- 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 258
- 2022Data Contamination: From Memorization to Exploitationcited by 236
- 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
- 2024LLM Dataset Inference: Did you train on my dataset?cited by 110
- 2024Blind Baselines Beat Membership Inference Attacks for Foundation Modelscited by 69
- 2016Mixture Proportion Estimation via Kernel Embedding of Distributionscited by 67
- 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoodscited by 64
- 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 59
- 2024Evading Data Contamination Detection for Language Models is (too) Easycited by 30
5 more
- 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
- 2024Towards Data Contamination Detection for Modern Large Language Models: Limitations, Inconsistencies, and Oracle Challengescited by 18
- 2025On The Fragility of Benchmark Contamination Detection in Reasoning Modelscited by 9
- 2026CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Modelscited by 3
- 2026The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detectioncited by 1
Built on by
Not cited yet.