# Search-Time Data Contamination arXiv: https://arxiv.org/abs/2508.13180 ยท HTML: https://mlsanity.com/paper/2508.13180/ ## Key facts - Cited by 19 papers in the map (data through 14 Oct 2026). - Builds on 14 of its 16 references that are in the map. - First posted on arXiv 12 Aug 2025. ## Builds on (most cited first, 14 of 14) - [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023) - [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022) - [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025) - [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025) - [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024) - [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025) - [A Careful Examination of Large Language Model Performance on Grade School Arithmetic](https://mlsanity.com/paper/2405.00332/) (2024) - [Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models](https://mlsanity.com/paper/2402.15938/) (2024) - [Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs](https://mlsanity.com/paper/2407.04694/) (2024) - [Search Arena: Analyzing Search-Augmented LLMs](https://mlsanity.com/paper/2506.05334/) (2025) - [BEARCUBS: A benchmark for computer-using web agents](https://mlsanity.com/paper/2503.07919/) (2025) - [Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediation](https://mlsanity.com/paper/2406.14644/) (2024) - [InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.15872/) (2025) - [Deep Research Comparator: A Platform For Fine-grained Human Annotations of Deep Research Agents](https://mlsanity.com/paper/2507.05495/) (2025) ## Built on by (most cited first, 19 of 19) - [SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents](https://mlsanity.com/paper/2509.06283/) (2025) - [ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge](https://mlsanity.com/paper/2510.18941/) (2025) - [MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes](https://mlsanity.com/paper/2510.16380/) (2025) - [MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents](https://mlsanity.com/paper/2601.12346/) (2026) - [Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Calling](https://mlsanity.com/paper/2509.26553/) (2025) - [Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wild](https://mlsanity.com/paper/2603.29020/) (2026) - [Synthesizing Agentic Data for Web Agents with Progressive Difficulty Enhancement Mechanisms](https://mlsanity.com/paper/2510.13913/) (2025) - [Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks](https://mlsanity.com/paper/2607.21763/) (2026) - [Quantifying the Effect of Test Set Contamination on Generative Evaluations](https://mlsanity.com/paper/2601.04301/) (2026) - [Reasoning Gets Harder for LLMs Inside A Dialogue](https://mlsanity.com/paper/2603.20133/) (2026) - [iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics](https://mlsanity.com/paper/2603.04656/) (2026) - [ToolAnchor: Anchoring Counterfactual Context to Boost Agentic Tool-use Capability](https://mlsanity.com/paper/2607.14145/) (2026) - [Eureka: Task-Conditioned Meta-Agent Orchestration for Scientific Discovery](https://mlsanity.com/paper/2608.19047/) (2026) - [Evaluating Deep-Search Agents under Hierarchical Web Evidence Poisoning](https://mlsanity.com/paper/2609.06027/) (2026) - [From Proof to Program: Characterizing Tool-Induced Reasoning Hallucinations in Large Language Models](https://mlsanity.com/paper/2511.10899/) (2025) - [Projecting BrowseComp-Plus onto ClimbMix: Toward More Realistic Corpora for Agentic Search](https://mlsanity.com/paper/2608.20317/) (2026) - [Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation](https://mlsanity.com/paper/2608.29463/) (2026) - [Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation](https://mlsanity.com/paper/2610.05830/) (2026) - [Avoiding What Your Teacher Already Knows: Entity Familiarity for Deep-Search Distillation](https://mlsanity.com/paper/2610.16251/) (2026) ## Abstract (arXiv) Data contamination refers to the leakage of evaluation data into model training data, resulting in overfitting to supposedly held-out test sets and compromising test validity. We identify an analogous issue, search-time contamination (STC), in evaluating search-based LLM agents which use tools to gather information from online sources when answering user queries. STC occurs when the retrieval step surfaces a source containing the test question (or a near-duplicate) alongside its answer, enabling agents to copy rather than genuinely infer or reason, undermining benchmark integrity. We find that HuggingFace, an online platform hosting evaluation datasets, appears among retrieved sources in search based agent logs. Consequently, agents often explicitly acknowledge discovering question answer pairs from HuggingFace within their reasoning chains. On three commonly used capability benchmarks: Humanity's Last Exam (HLE), SimpleQA, and GPQA, we demonstrate that for approximately 3% of questions, search-based agents directly find the datasets with ground truth labels on HuggingFace. When millions of evaluation queries target the same benchmark, even small, repeated leaks can accelerate the benchmark's obsolescence, shortening its intended lifecycle. After HuggingFace is blocked, we observe a drop in accuracy on the contaminated subset of approximately 15%. We further show through ablation experiments that publicly accessible evaluation datasets on HuggingFace may not be the sole source of STC. To this end, we conclude by proposing best practices for benchmark design and result reporting to address this novel form of leakage and ensure trustworthy evaluation of search-based LLM agents. To facilitate the auditing of evaluation results, we also publicly release the complete logs from our experiments.