# On Inter-dataset Code Duplication and Data Leakage in Large Language Models

arXiv: https://arxiv.org/abs/2401.07930 · HTML: https://mlsanity.com/paper/2401.07930/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jan 2024.

## Built on by (most cited first, 10 of 10)

- [Cofca: A Step-Wise Counterfactual Multi-hop QA benchmark](https://mlsanity.com/paper/2402.11924/) (2024)
- [Evaluating the Generalizability of LLMs in Automated Program Repair](https://mlsanity.com/paper/2503.09217/) (2025)
- [Trust Me, I Know This Function: Hijacking LLM Static Analysis using Bias](https://mlsanity.com/paper/2508.17361/) (2025)
- [CGP-Tuning: Structure-Aware Soft Prompt Tuning for Code Vulnerability Detection](https://mlsanity.com/paper/2501.04510/) (2025)
- [The Ever-Evolving Science Exam](https://mlsanity.com/paper/2507.16514/) (2025)
- [Beyond Retrieval: A Multitask Benchmark and Model for Code Search](https://mlsanity.com/paper/2605.04615/) (2026)
- [Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Models](https://mlsanity.com/paper/2507.17389/) (2025)
- [From Data Leak to Secret Misses: The Impact of Data Leakage on Secret Detection Models](https://mlsanity.com/paper/2601.22946/) (2026)
- [Enhancing Continual Learning for Software Vulnerability Prediction: Addressing Catastrophic Forgetting via Hybrid-Confidence-Aware Selective Replay for Temporal LLM Fine-Tuning](https://mlsanity.com/paper/2602.23834/) (2026)
- [A Metamorphic Testing Approach to Diagnosing Memorization in LLM-Based Program Repair](https://mlsanity.com/paper/2604.21579/) (2026)

## Abstract (arXiv)

Motivation. Large language models (LLMs) have exhibited remarkable proficiency in diverse software engineering (SE) tasks. Handling such tasks typically involves acquiring foundational coding knowledge on large, general-purpose datasets during a pre-training phase, and subsequently refining on smaller, task-specific datasets as part of a fine-tuning phase. Problem statement. While intra-dataset code duplication examines the intersection between the training and test splits within a given dataset and has been addressed in prior research, inter-dataset code duplication, which gauges the overlap between different datasets, remains largely unexplored. If this phenomenon exists, it could compromise the integrity of LLM evaluations because of the inclusion of fine-tuning test samples that were already encountered during pre-training, resulting in inflated performance metrics. Contribution. This paper explores the phenomenon of inter-dataset code duplication and its impact on evaluating LLMs across diverse SE tasks. Study design. We conduct an empirical study using the CodeSearchNet dataset (CSN), a widely adopted pre-training dataset, and five fine-tuning datasets used for various se tasks. We first identify the intersection between the pre-training and fine-tuning datasets using a deduplication process. Next, we pre-train two versions of LLMs using a subset of CSN: one leaky LLM and one non-leaky LLM. Finally, we fine-tune both models and compare their performances using leaky fine-tuning test samples. Results. Our findings reveal a potential threat to the evaluation of LLMs across multiple SE tasks, stemming from the inter-dataset code duplication phenomenon. We also demonstrate that this threat is accentuated by the chosen fine-tuning technique. Furthermore, we provide evidence that open-source models could be affected by inter-dataset duplication.