# CLEAN-EVAL: Clean Evaluation on Contaminated Large Language Models

arXiv: https://arxiv.org/abs/2311.09154 · HTML: https://mlsanity.com/paper/2311.09154/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Skywork: A More Open Bilingual Foundation Model](https://mlsanity.com/paper/2310.19341/) (2023)
- [ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks](https://mlsanity.com/paper/2310.02569/) (2023)

## Built on by (most cited first, 13 of 13)

- [Benchmarking Benchmark Leakage in Large Language Models](https://mlsanity.com/paper/2404.18824/) (2024)
- [ConStat: Performance-Based Contamination Detection in Large Language Models](https://mlsanity.com/paper/2405.16281/) (2024)
- [Evading Data Contamination Detection for Language Models is (too) Easy](https://mlsanity.com/paper/2402.02823/) (2024)
- [Cofca: A Step-Wise Counterfactual Multi-hop QA benchmark](https://mlsanity.com/paper/2402.11924/) (2024)
- [Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model](https://mlsanity.com/paper/2410.18640/) (2024)
- [The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination](https://mlsanity.com/paper/2503.16402/) (2025)
- [A Comprehensive Survey of Contamination Detection Methods in Large Language Models](https://mlsanity.com/paper/2404.00699/) (2024)
- [DICE: Detecting In-distribution Contamination in LLM's Fine-tuning Phase for Math Reasoning](https://mlsanity.com/paper/2406.04197/) (2024)
- [Reliable and diverse evaluation of LLM medical knowledge mastery](https://mlsanity.com/paper/2409.14302/) (2024)
- [LogitTrace: Detecting Benchmark Contamination via Layerwise Logit Trajectories](https://mlsanity.com/paper/2509.20909/) (2025)
- [AdEval: Alignment-based Dynamic Evaluation to Mitigate Data Contamination in Large Language Models](https://mlsanity.com/paper/2501.13983/) (2025)
- [Provable Training Data Identification for Large Language Models](https://mlsanity.com/paper/2510.09717/) (2025)
- [Provable Joint Decontamination for Benchmarking Multiple Large Language Models](https://mlsanity.com/paper/2605.21543/) (2026)

## Abstract (arXiv)

We are currently in an era of fierce competition among various large language models (LLMs) continuously pushing the boundaries of benchmark performance. However, genuinely assessing the capabilities of these LLMs has become a challenging and critical issue due to potential data contamination, and it wastes dozens of time and effort for researchers and engineers to download and try those contaminated models. To save our precious time, we propose a novel and useful method, Clean-Eval, which mitigates the issue of data contamination and evaluates the LLMs in a cleaner manner. Clean-Eval employs an LLM to paraphrase and back-translate the contaminated data into a candidate set, generating expressions with the same meaning but in different surface forms. A semantic detector is then used to filter the generated low-quality samples to narrow down this candidate set. The best candidate is finally selected from this set based on the BLEURT score. According to human assessment, this best candidate is semantically similar to the original contamination data but expressed differently. All candidates can form a new benchmark to evaluate the model. Our experiments illustrate that Clean-Eval substantially restores the actual evaluation results on contaminated LLMs under both few-shot learning and fine-tuning scenarios.