# TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMs

arXiv: https://arxiv.org/abs/2605.24079 · HTML: https://mlsanity.com/paper/2605.24079/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 50 references that are in the map.
- First posted on arXiv 22 May 2026.

## Builds on (most cited first, 30 of 35)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Magicoder: Empowering Code Generation with OSS-Instruct](https://mlsanity.com/paper/2312.02120/) (2023)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://mlsanity.com/paper/2311.04850/) (2023)
- [Do Membership Inference Attacks Work on Large Language Models?](https://mlsanity.com/paper/2402.07841/) (2024)
- [Platypus: Quick, Cheap, and Powerful Refinement of LLMs](https://mlsanity.com/paper/2308.07317/) (2023)
- [Benchmark Data Contamination of Large Language Models: A Survey](https://mlsanity.com/paper/2406.04244/) (2024)
- [Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks](https://mlsanity.com/paper/2305.10160/) (2023)
- [jina-embeddings-v3: Multilingual Embeddings With Task LoRA](https://mlsanity.com/paper/2409.10173/) (2024)
- [Investigating Data Contamination in Modern Benchmarks for Large Language Models](https://mlsanity.com/paper/2311.09783/) (2023)
- [Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models](https://mlsanity.com/paper/2402.15938/) (2024)
- [Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models](https://mlsanity.com/paper/2404.02936/) (2024)
- [Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models](https://mlsanity.com/paper/2403.04811/) (2024)
- [LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction](https://mlsanity.com/paper/2312.12343/) (2023)
- [SourcererCC: Scaling Code Clone Detection to Big Code](https://mlsanity.com/paper/1512.06448/) (2015)
- [Paraphrasing with Large Language Models](https://mlsanity.com/paper/1911.09661/) (2019)
- [A Survey on Data Contamination for Large Language Models](https://mlsanity.com/paper/2502.14425/) (2025)
- [The ROOTS Search Tool: Data Transparency for LLMs](https://mlsanity.com/paper/2302.14035/) (2023)
- [Evading Data Contamination Detection for Language Models is (too) Easy](https://mlsanity.com/paper/2402.02823/) (2024)
- [Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions](https://mlsanity.com/paper/2410.18966/) (2024)
- [TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs](https://mlsanity.com/paper/2403.00393/) (2024)

## Built on by (most cited first, 1 of 1)

- [MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses](https://mlsanity.com/paper/2609.33411/) (2026)

## Abstract (arXiv)

Data contamination is a known threat to the reliability of model evaluation. However, it remains underexplored in code large language models (LLMs), where contamination often goes beyond exact duplication. We present TRACER, a semantic-aware framework for fine-grained code contamination detection. TRACER models contamination using three levels of semantic overlap - Functionally Identical, Nearly Identical, and Shared Logic - and detects them through a coarse-to-fine pipeline. We also introduce the first benchmark for fine-grained code contamination detection, spanning three widely used benchmarks and three representative post-training datasets. TRACER achieves strong and consistent performance across multiple LLM backbones, with GPT-5 reaching an F1 score of 0.91 in fine-grained detection. In the binary setting, TRACER attains an F1 of 0.92, outperforming existing methods by 42%-217%. We further conduct ablation studies and error analysis to assess the contributions of individual components in TRACER.