# From Data Leak to Secret Misses: The Impact of Data Leakage on Secret Detection Models

arXiv: https://arxiv.org/abs/2601.22946 · HTML: https://mlsanity.com/paper/2601.22946/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Builds on (most cited first, 12 of 12)

- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018)
- [Dos and Don'ts of Machine Learning in Computer Security](https://mlsanity.com/paper/2010.09470/) (2020)
- [LLM Security Guard for Code](https://mlsanity.com/paper/2405.01103/) (2024)
- [Chasing Shadows: Pitfalls in LLM Security Research](https://mlsanity.com/paper/2512.09549/) (2025)
- [Benchmarking Prompt Engineering Techniques for Secure Code Generation with GPT Models](https://mlsanity.com/paper/2502.06039/) (2025)
- [On Inter-dataset Code Duplication and Data Leakage in Large Language Models](https://mlsanity.com/paper/2401.07930/) (2024)
- [SecretBench: A Dataset of Software Secrets](https://mlsanity.com/paper/2303.06729/) (2023)
- [Secret Breach Detection in Source Code with Large Language Models](https://mlsanity.com/paper/2504.18784/) (2025)
- [AssetHarvester: A Static Analysis Tool for Detecting Secret-Asset Pairs in Software Artifacts](https://mlsanity.com/paper/2403.19072/) (2024)
- [Security Bug Report Prediction Within and Across Projects: A Comparative Study of BERT and Random Forest](https://mlsanity.com/paper/2504.21037/) (2025)

## Abstract (arXiv)

Machine learning models are increasingly used for software security tasks. These models are commonly trained and evaluated on large Internet-derived datasets, which often contain duplicated or highly similar samples. When such samples are split across training and test sets, data leakage may occur, allowing models to memorize patterns instead of learning to generalize. We investigate duplication in a widely used benchmark dataset of hard coded secrets and show how data leakage can substantially inflate the reported performance of AI-based secret detectors, resulting in a misleading picture of their real-world effectiveness.