mlsanityData to 7 Oct 2026

LeakScale: Estimating the Causal Effect of Benchmark Exposure

arXiv 2609.27176 ↗First posted 23 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Evidence that evaluation material entered training does not reveal how much it affected evaluation. This distinction leaves a contaminated benchmark score difficult to interpret: provenance can establish contact, but only a counterfactual can quantify the performance attributable to that contact. We present LeakScale, an interventional framework for estimating this missing quantity. LeakScale creates fresh executable tasks that require private, family-specific information absent from and non-derivable from the public task, controls access to that information, and estimates the resulting control-adjusted change in executable accuracy. Across 2,048 unique families, two model families, two executable domains, and 262,144 generations, exposure improves accuracy in every model-by-domain combination, with gains ranging from +7.17 to +27.31 percentage points. These findings separate two empirical questions that are often conflated: whether benchmark contact occurred and how strongly a reported score depends on it. LeakScale makes the latter directly measurable.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 29 references that are in the map. First posted on arXiv 23 Sep 2026.

Builds on

26 of 29 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  6. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  7. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  8. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  9. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  10. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  11. 2026Gemma 4 Technical Reportcited by 187
  12. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
14 more
  1. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  2. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136
  3. 2023An Open Source Data Contamination Report for Large Language Modelscited by 47
  4. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  5. 2024Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediationcited by 30
  6. 2024Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?cited by 30
  7. 2024Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluationcited by 18
  8. 2025Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translationcited by 13
  9. 2025On The Fragility of Benchmark Contamination Detection in Reasoning Modelscited by 8
  10. 2024Fine-tuning can Help Detect Pretraining Data from Large Language Modelscited by 7
  11. 2025CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfittingcited by 6
  12. 2025Is Your Benchmark Still Useful? Dynamic Benchmarking for Code Language Modelscited by 5
  13. 2025Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Modelscited by 4
  14. 2025DyePack: Provably Flagging Test Set Contamination in LLMs Using Backdoorscited by 3

Built on by

Not cited yet.