# Your Code Secret Belongs to Me: Neural Code Completion Tools Can Memorize Hard-Coded Credentials

arXiv: https://arxiv.org/abs/2309.07639 · HTML: https://mlsanity.com/paper/2309.07639/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Sep 2023.

## Built on by (most cited first, 17 of 17)

- [SoK: Memorization in General-Purpose Large Language Models](https://mlsanity.com/paper/2310.18362/) (2023)
- [Undesirable Memorization in Large Language Models: A Survey](https://mlsanity.com/paper/2410.02650/) (2024)
- [Security Attacks on LLM-based Code Completion Tools](https://mlsanity.com/paper/2408.11006/) (2024)
- ["Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editors](https://mlsanity.com/paper/2509.22040/) (2025)
- [Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey](https://mlsanity.com/paper/2310.17903/) (2023)
- [Scrub It Out! Erasing Sensitive Memorization in Code Language Models via Machine Unlearning](https://mlsanity.com/paper/2509.13755/) (2025)
- [Promise and Peril of Collaborative Code Generation Models: Balancing Effectiveness and Memorization](https://mlsanity.com/paper/2409.12020/) (2024)
- [LibreLog: Accurate and Efficient Unsupervised Log Parsing Using Open-Source Large Language Models](https://mlsanity.com/paper/2408.01585/) (2024)
- [Contextualized Data-Wrangling Code Generation in Computational Notebooks](https://mlsanity.com/paper/2409.13551/) (2024)
- [Next Edit Prediction: Learning to Predict Code Edits from Context and Interaction History](https://mlsanity.com/paper/2508.10074/) (2025)
- [Mitigating Sensitive Information Leakage in LLMs4Code through Machine Unlearning](https://mlsanity.com/paper/2502.05739/) (2025)
- [Effective Code Membership Inference for Code Completion Models via Adversarial Prompts](https://mlsanity.com/paper/2511.15107/) (2025)
- [Understanding Privacy Risks in Code Models Through Training Dynamics: A Causal Approach](https://mlsanity.com/paper/2512.07814/) (2025)
- [Malicious and Unintentional Disclosure Risks in Large Language Models for Code Generation](https://mlsanity.com/paper/2503.22760/) (2025)
- [Towards Privacy-Preserving Code Generation: Differentially Private Code Language Models](https://mlsanity.com/paper/2512.11482/) (2025)
- [Protecting Private Code in IDE Autocomplete using Differential Privacy](https://mlsanity.com/paper/2601.22935/) (2026)
- [Understanding Secret Leakage Risks in Code LLMs: A Tokenization Perspective](https://mlsanity.com/paper/2604.17814/) (2026)

## Abstract (arXiv)

Neural Code Completion Tools (NCCTs) have reshaped the field of software engineering, which are built upon the language modeling technique and can accurately suggest contextually relevant code snippets. However, language models may emit the training data verbatim during inference with appropriate prompts. This memorization property raises privacy concerns of NCCTs about hard-coded credential leakage, leading to unauthorized access to applications, systems, or networks. Therefore, to answer whether NCCTs will emit the hard-coded credential, we propose an evaluation tool called Hard-coded Credential Revealer (HCR). HCR constructs test prompts based on GitHub code files with credentials to reveal the memorization phenomenon of NCCTs. Then, HCR designs four filters to filter out ill-formatted credentials. Finally, HCR directly checks the validity of a set of non-sensitive credentials. We apply HCR to evaluate three representative types of NCCTs: Commercial NCCTs, open-source models, and chatbots with code completion capability. Our experimental results show that NCCTs can not only return the precise piece of their training data but also inadvertently leak additional secret strings. Notably, two valid credentials were identified during our experiments. Therefore, HCR raises a severe privacy concern about the potential leakage of hard-coded credentials in the training data of commercial NCCTs. All artifacts and data are released for future research purposes in https://github.com/HCR-Repo/HCR.