Silicon Bureaucracy and AI Test-Oriented Education: Contamination Sensitivity and Score Confidence in LLM Benchmarks
Not on the map. Why?
Abstract (arXiv)
Public benchmarks increasingly govern how large language models (LLMs) are ranked, selected, and deployed. We frame this benchmark-centered regime as Silicon Bureaucracy and AI Test-Oriented Education, and argue that it rests on a fragile assumption: that benchmark scores directly reflect genuine generalization. In practice, however, such scores may conflate exam-oriented competence with principled capability, especially when contamination and semantic leakage are difficult to exclude from modern training pipelines. We therefore propose an audit framework for analyzing contamination sensitivity and score confidence in LLM benchmarks. Using a router-worker setup, we compare a clean-control condition with noisy conditions in which benchmark problems are systematically deleted, rewritten, and perturbed before being passed downstream. For a genuinely clean benchmark, noisy conditions should not consistently outperform the clean-control baseline. Yet across multiple models, we find widespread but heterogeneous above-baseline gains under noisy conditions, indicating that benchmark-related cues may be reassembled and can reactivate contamination-related memory. These results suggest that similar benchmark scores may carry substantially different levels of confidence. Rather than rejecting benchmarks altogether, we argue that benchmark-based evaluation should be supplemented with explicit audits of contamination sensitivity and score confidence.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 33 references that are in the map. First posted on arXiv 23 Mar 2026.
Builds on
27 of 33 references.
References · 27
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
- 2025Humanity's Last Examcited by 467
- 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
- 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
- 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
- 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
- 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 112
- 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
- 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Usecited by 48
- 2025AI Flow: Perspectives, Scenarios, and Approachescited by 44
- 2024AI Flow at the Network Edgecited by 43
- 2023PromptBench: A Unified Library for Evaluation of Large Language Modelscited by 42
15 more
- 2024ConStat: Performance-Based Contamination Detection in Large Language Modelscited by 33
- 2024DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graphcited by 26
- 2024Training on the Benchmark Is Not All You Needcited by 24
- 2025ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoningcited by 17
- 2023M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Modelscited by 15
- 2024TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planningcited by 14
- 2025The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contaminationcited by 13
- 2025AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Researchcited by 12
- 2025UQLM: A Python Package for Uncertainty Quantification in Large Language Modelscited by 9
- 2024BenchmarkCards: Standardized Documentation for Large Language Model Benchmarkscited by 8
- 2025MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Modelscited by 7
- 2025Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compressioncited by 7
- 2025Theoretical Foundations of Scaling Law in Familial Modelscited by 3
- 2026Ruyi2 Technical Reportcited by 2
- 2026Are LLMs Vulnerable to Preference-Undermining Attacks (PUA)? A Factorial Analysis Methodology for Diagnosing the Trade-off between Preference Alignment and Real-World Validitycited by 1
Built on by
Not cited yet.