mlsanityData to 7 Oct 2026

AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge

arXiv 2412.13670 ↗First posted 18 Dec 2024cs.CL cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

Data contamination hinders fair LLM evaluation by introducing test data into newer models' training sets. Existing studies solve this challenge by updating benchmarks with newly collected data. However, they fail to guarantee contamination-free evaluation as the newly collected data may contain pre-existing knowledge, and their benchmark updates rely on intensive human labor. To address these issues, we in this paper propose AntiLeak-Bench, an automated anti-leakage benchmarking framework. Instead of simply using newly collected data, we construct samples with explicitly new knowledge absent from LLMs' training sets, which thus ensures strictly contamination-free evaluation. We further design a fully automated workflow to build and update our benchmark without human labor. This significantly reduces the cost of benchmark maintenance to accommodate emerging LLMs. Through extensive experiments, we highlight that data contamination likely exists before LLMs' cutoff time and demonstrate AntiLeak-Bench effectively overcomes this challenge.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 40 of its 61 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

40 of 61 references.

References · 40

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2024Qwen2 Technical Reportcited by 2,076
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
28 more
  1. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  2. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  3. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  4. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292
  5. 2022RealTime QA: What's the Answer Right Now?cited by 239
  6. 2022Data Contamination: From Memorization to Exploitationcited by 234
  7. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  8. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  9. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  10. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  11. 2023Fact-Checking Complex Claims with Program-Guided Reasoningcited by 146
  12. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  13. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 123
  14. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  15. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  16. 2020DynaSent: A Dynamic Benchmark for Sentiment Analysiscited by 63
  17. 2023Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluationcited by 52
  18. 2022Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learningcited by 38
  19. 2024AKEW: Assessing Knowledge Editing in the Wildcited by 35
  20. 2024FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Modelcited by 34
  21. 2024RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarioscited by 26
  22. 2023InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic Modelingcited by 24
  23. 2024Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmarkcited by 22
  24. 2023Towards the TopMost: A Topic Modeling System Toolkitcited by 21
  25. 2024On the Affinity, Rationality, and Diversity of Hierarchical Topic Modelingcited by 18
  26. 2022Dynatask: A Framework for Creating Dynamic AI Benchmark Taskscited by 16
  27. 2024Modeling Dynamic Topics in Chain-Free Fashion by Evolution-Tracking Contrastive Learning and Unassociated Word Exclusioncited by 16
  28. 2024Are LLMs Good Zero-Shot Fallacy Classifiers?cited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 22; 2026: 19.

42 papers.

Sort:

By area:

  1. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  2. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  3. 2025Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddingscited by 26
  4. 2024MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answeringcited by 17
  5. 2025Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoningcited by 14
  6. 2025LastingBench: Defend Benchmarks Against Knowledge Leakagecited by 14
  7. 2025The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contaminationcited by 13
  8. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 6
  9. 2025Unsupervised Hallucination Detection by Inspecting Reasoning Processescited by 5
  10. 2025CutPaste&Find: Efficient Multimodal Hallucination Detector with Visual-aid Knowledge Basecited by 4
  11. 2025DyePack: Provably Flagging Test Set Contamination in LLMs Using Backdoorscited by 3
  12. 2025Lean Meets Theoretical Computer Science: Scalable Synthesis of Theorem Proving Challenges in Formal-Informal Pairscited by 3
30 more
  1. 2025Test of Time: Rethinking Temporal Signal of Benchmark Contaminationcited by 3
  2. 2025VCBench: Benchmarking LLMs in Venture Capitalcited by 3
  3. 2025CoreEval: Automatically Building Contamination-Resilient Datasets with Real-World Knowledge toward Reliable LLM Evaluationcited by 3
  4. 2025LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligencecited by 3
  5. 2025Aspect-Based Summarization with Self-Aspect Retrieval Enhanced Generationcited by 2
  6. 2025The Ever-Evolving Science Examcited by 2
  7. 2026LLM Benchmark Datasets Should Be Contamination-Resistantcited by 2
  8. 2026ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgmentcited by 2
  9. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 2
  10. 2025SCOPE: Compress Mathematical Reasoning Steps for Efficient Automated Process Annotationcited by 1
  11. 2025MatheMagic: Generating Dynamic Mathematics Benchmarks Robust to Memorizationcited by 1
  12. 2025Individualized Cognitive Simulation in Large Language Models: Evaluating Different Cognitive Representation Methodscited by 1
  13. 2025DynaSolidGeo: A Dynamic Benchmark for Genuine Spatial Mathematical Reasoning of VLMs in Solid Geometrycited by 1
  14. 2026How Do Large Language Models Learn Concepts During Continual Pre-Training?cited by 1
  15. 2026Can David Beat Goliath? On Multi-Hop Reasoning with Resource-Constrained Agentscited by 1
  16. 2026From 0-Order Selection to 2-Order Judgment: Combinatorial Hardening Exposes Compositional Failures in Frontier LLMscited by 1
  17. 2026Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implicationscited by 1
  18. 2026GDPevo: Evaluating Agent Self-Evolution on Real Business Taskscited by 1
  19. 2025EMERGE: A Benchmark for Updating Knowledge Graphs with Emerging Textual Knowledgecited by 0
  20. 2025Harnessing Temporal Databases for Systematic Evaluation of Factual Time-Sensitive Question-Answering in Large Language Modelscited by 0
  21. 2026GLEAN: Grounded Lightweight Evaluation Anchors for Contamination-Aware Tabular Reasoningcited by 0
  22. 2026Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilitiescited by 0
  23. 2026ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?cited by 0
  24. 2026Uncertainty-based Debiasing and Unlearning for Decontaminationcited by 0
  25. 2026MMBench-Live: A Continuously Evolving Benchmark for Multimodal Modelscited by 0
  26. 2026Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scorescited by 0
  27. 2026Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcutscited by 0
  28. 2026Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Modelscited by 0
  29. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0
  30. 2026WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnessescited by 0