mlsanityData to 14 Oct 2026

LastingBench: Defend Benchmarks Against Knowledge Leakage

arXiv 2506.21614 ↗First posted 21 Jun 2025cs.CL cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

The increasing complexity of large language models (LLMs) raises concerns about their ability to "cheat" on standard Question Answering (QA) benchmarks by memorizing task-specific data. This undermines the validity of benchmark evaluations, as they no longer reflect genuine model capabilities but instead the effects of data leakage. While prior work has focused on detecting such leakage, little attention has been given to mitigating its impact and preserving the long-term utility of benchmarks. In this paper, we introduce LastingBench, a novel framework designed to continuously reinforce and safeguard existing benchmarks against knowledge leakage. LastingBench identifies leakage points in the context through perturbation, then rewrites the leakage points to counterfactual ones-disrupting memorization while preserving the benchmark's original evaluative intent. Evaluations of state-of-the-art QA benchmarks show significant performance gaps, highlighting the efficacy of LastingBench in reducing memorization effects. LastingBench offers a practical and scalable solution to ensure benchmark robustness over time, promoting fairer and more interpretable evaluations of LLMs.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 40 of its 57 references that are in the map. First posted on arXiv 21 Jun 2025.

Builds on

40 of 57 references.

References · 40

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2020Extracting Training Data from Large Language Modelscited by 2,189
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  9. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  10. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  11. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
  12. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
28 more
  1. 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
  2. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  3. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  4. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  5. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  6. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 126
  7. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 125
  8. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  9. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 119
  10. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 81
  11. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  12. 2025ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agentscited by 66
  13. 2025GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoningcited by 58
  14. 2024AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledgecited by 44
  15. 2024Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Accelerationcited by 39
  16. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  17. 2025A Survey on Data Contamination for Large Language Modelscited by 35
  18. 2024Long2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recallcited by 32
  19. 2024Evading Data Contamination Detection for Language Models is (too) Easycited by 30
  20. 2024VarBench: Robust Language Model Benchmarking Through Dynamic Variable Perturbationcited by 29
  21. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  22. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  23. 2024RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructionscited by 25
  24. 2024TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMscited by 24
  25. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervisioncited by 21
  26. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystackcited by 20
  27. 2025ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programmingcited by 16
  28. 2024ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coveragecited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 11.

14 papers.

Sort:

By area:

  1. 2026SWE-Pruner: Self-Adaptive Context Pruning for Coding Agentscited by 44
  2. 2025Latent Inter-User Difference Modeling for LLM Personalizationcited by 21
  3. 2026Reasoning in Trees: Improving Retrieval-Augmented Generation for Multi-Hop Question Answeringcited by 14
  4. 2026Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understandingcited by 13
  5. 2026CTFusion: A CTF-based Benchmark for LLM Agent Evaluationcited by 4
  6. 2026EarlyEval: Cheaper Agent Evaluation via Early Outcome Predictioncited by 3
  7. 2026ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generationcited by 2
  8. 2026SWE-Pruner Pro: The Coder LLM Already Knows What to Prunecited by 2
  9. 2025CANDY: Benchmarking LLMs' Limitations and Assistive Potential in Chinese Misinformation Fact-Checkingcited by 1
  10. 2026QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarkscited by 1
  11. 2025Causal Invariance and Counterfactual Learning Driven Cooperative Game for Multi-Label Classificationcited by 0
  12. 2026When Fusion Helps and When It Breaks: View-Aligned Robustness in Same-Source Financial Imagingcited by 0
2 more
  1. 2026GLEAN: Grounded Lightweight Evaluation Anchors for Contamination-Aware Tabular Reasoningcited by 0
  2. 2026Schr"odinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarking Benchmark Leakage in Large Language Models
  2. 2026Generating Leakage-Free Benchmarks for Robust RAG Evaluation
  3. 2025Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements
  4. 2025LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge
  5. 2025Test of Time: Rethinking Temporal Signal of Benchmark Contamination
  6. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progress
  7. 2026The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning
  8. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models