mlsanityData to 7 Oct 2026

Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation

arXiv 2502.17521 ↗First posted 23 Feb 2025cs.LG cs.AI cs.CLcited by 38

Not on the map. Why?

Abstract (arXiv)

Data contamination has received increasing attention in the era of large language models (LLMs) due to their reliance on vast Internet-derived training corpora. To mitigate the risk of potential data contamination, LLM benchmarking has undergone a transformation from static to dynamic benchmarking. In this work, we conduct an in-depth analysis of existing static to dynamic benchmarking methods aimed at reducing data contamination risks. We first examine methods that enhance static benchmarks and identify their inherent limitations. We then highlight a critical gap-the lack of standardized criteria for evaluating dynamic benchmarks. Based on this observation, we propose a series of optimal design principles for dynamic benchmarking and analyze the limitations of existing dynamic benchmarks. This survey provides a concise yet comprehensive overview of recent advancements in data contamination research, offering valuable insights and a clear guide for future research efforts. We maintain a GitHub repository to continuously collect both static and dynamic benchmarking methods for LLMs. The repository can be found at this link.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 87 of its 118 references that are in the map. First posted on arXiv 23 Feb 2025.

Builds on

87 of 118 references.

References · 87

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
38 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  13. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  14. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  15. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  16. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  17. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  18. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  19. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  20. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  21. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  22. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  23. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  24. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  25. 2018QuAC : Question Answering in Contextcited by 379
  26. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  27. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  28. 2022Data Contamination: From Memorization to Exploitationcited by 234
  29. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  30. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  31. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  32. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  33. 2024Are We Done with MMLU?cited by 178
  34. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
  35. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  36. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  37. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  38. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 128

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 19.

38 papers.

Sort:

By area:

  1. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  2. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  3. 2025Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contaminationcited by 22
  4. 2025ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problemscited by 17
  5. 2025LastingBench: Defend Benchmarks Against Knowledge Leakagecited by 14
  6. 2026Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJackcited by 13
  7. 2025TreeDiff: AST-Guided Code Generation with Diffusion LLMscited by 11
  8. 2025NetArena: Dynamic Benchmarks for AI Agents in Network Automationcited by 8
  9. 2025VAR-MATH: Probing True Mathematical Reasoning in LLMS via Symbolic Multi-Instance Benchmarkscited by 8
  10. 2025BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Modelscited by 7
  11. 2025CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfittingcited by 6
  12. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
26 more
  1. 2025Jailbreak Distillation: Renewable Safety Benchmarkingcited by 4
  2. 2025Enhancing Test-Time Scaling of Large Language Models with Hierarchical Retrieval-Augmented MCTScited by 4
  3. 2026Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoningcited by 4
  4. 2025Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corporacited by 2
  5. 2025Is General-Purpose AI Reasoning Sensitive to Data-Induced Cognitive Biases? Dynamic Benchmarking on Typical Software Engineering Dilemmascited by 2
  6. 2026LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detectioncited by 2
  7. 2026CFMS: A Coarse-to-Fine Multimodal Synthesis Framework for Enhanced Tabular Reasoningcited by 2
  8. 2026CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarkscited by 2
  9. 2025ArenaBencher: Automatic Benchmark Evolution via Multi-Model Competitive Evaluationcited by 1
  10. 2025DynaSolidGeo: A Dynamic Benchmark for Genuine Spatial Mathematical Reasoning of VLMs in Solid Geometrycited by 1
  11. 2026Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implicationscited by 1
  12. 2026Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Modelcited by 1
  13. 2025Answer-Centric or Reasoning-Driven? Uncovering the Latent Memory Anchor in LLMscited by 0
  14. 2025ReaLM: Reflection-Enhanced Autonomous Reasoning with Small Language Modelscited by 0
  15. 2025Financial Instruction Following Evaluation (FIFE)cited by 0
  16. 2026Query Disambiguation via Answer-Free Context: Doubling Performance on Humanity's Last Examcited by 0
  17. 2026DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Modelscited by 0
  18. 2026STELLAR-E: a Synthetic, Tailored, End-to-end LLM Application Rigorous Evaluatorcited by 0
  19. 2026REKEY: Metadata-Grounded Visual-Key Regeneration for Contamination-Resilient VQA Evaluationcited by 0
  20. 2026Power Systems Agent Benchmark: Executable Evaluation of AI Agents in Electric Power Engineeringcited by 0
  21. 2026StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMscited by 0
  22. 2026Meta-Benchmarks for Financial-Services LLM Evaluationcited by 0
  23. 2026A2utoLPBench: An Auto-Generated, Agent-Friendly LP Benchmark via Inverse-KKT Constructioncited by 0
  24. 2026ALPS: Measuring Valid Creativity in Large Language Models with Mathematical Constructioncited by 0
  25. 2026LeakScale: Estimating the Causal Effect of Benchmark Exposurecited by 0
  26. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0