# Security Attacks on LLM-based Code Completion Tools

arXiv: https://arxiv.org/abs/2408.11006 · HTML: https://mlsanity.com/paper/2408.11006/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 20 Aug 2024.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models](https://mlsanity.com/paper/2310.02949/) (2023)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks](https://mlsanity.com/paper/2302.05733/) (2023)
- [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023)
- [Inverse Scaling: When Bigger Isn't Better](https://mlsanity.com/paper/2306.09479/) (2023)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning](https://mlsanity.com/paper/2212.08061/) (2022)
- [MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots](https://mlsanity.com/paper/2307.08715/) (2023)
- [LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?](https://mlsanity.com/paper/2307.10719/) (2023)
- [Your Code Secret Belongs to Me: Neural Code Completion Tools Can Memorize Hard-Coded Credentials](https://mlsanity.com/paper/2309.07639/) (2023)
- [Demystifying Practices, Challenges and Expected Features of Using GitHub Copilot](https://mlsanity.com/paper/2309.05687/) (2023)

## Built on by (most cited first, 16 of 16)

- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [A Survey on Trustworthy LLM Agents: Threats and Countermeasures](https://mlsanity.com/paper/2503.09648/) (2025)
- [Large Language Models for Computer-Aided Design: A Survey](https://mlsanity.com/paper/2505.08137/) (2025)
- [Large Language Models (LLMs) for Source Code Analysis: applications, models and datasets](https://mlsanity.com/paper/2503.17502/) (2025)
- [When Lower Privileges Suffice: Investigating Over-Privileged Tool Selection in LLM Agents](https://mlsanity.com/paper/2606.20023/) (2026)
- ["X of Information'' Continuum: A Survey on AI-Driven Multi-dimensional Metrics for Next-Generation Networked Systems](https://mlsanity.com/paper/2507.19657/) (2025)
- [Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks](https://mlsanity.com/paper/2510.01359/) (2025)
- [From Completion to Editing: Unlocking Context-Aware Code Infilling via Search-and-Replace Instruction Tuning](https://mlsanity.com/paper/2601.13384/) (2026)
- [AgentRaft: Automated Detection of Data Over-Exposure in LLM Agents](https://mlsanity.com/paper/2603.07557/) (2026)
- [ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generation](https://mlsanity.com/paper/2509.07941/) (2025)
- [How Secure is Secure Code Generation? Adversarial Prompts Put LLM Defenses to the Test](https://mlsanity.com/paper/2601.07084/) (2026)
- [SoK: Understanding (New) Security Issues Across AI4Code Use Cases](https://mlsanity.com/paper/2512.18456/) (2025)
- [Protecting Private Code in IDE Autocomplete using Differential Privacy](https://mlsanity.com/paper/2601.22935/) (2026)
- [Micro Language Models Enable Instant Responses](https://mlsanity.com/paper/2604.19642/) (2026)
- [Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code](https://mlsanity.com/paper/2606.11817/) (2026)
- [The Illusion of Secure LLM Code: Closing the Security Gap via Iterative Reprompting](https://mlsanity.com/paper/2607.23710/) (2026)

## Abstract (arXiv)

The rapid development of large language models (LLMs) has significantly advanced code completion capabilities, giving rise to a new generation of LLM-based Code Completion Tools (LCCTs). Unlike general-purpose LLMs, these tools possess unique workflows, integrating multiple information sources as input and prioritizing code suggestions over natural language interaction, which introduces distinct security challenges. Additionally, LCCTs often rely on proprietary code datasets for training, raising concerns about the potential exposure of sensitive data. This paper exploits these distinct characteristics of LCCTs to develop targeted attack methodologies on two critical security risks: jailbreaking and training data extraction attacks. Our experimental results expose significant vulnerabilities within LCCTs, including a 99.4% success rate in jailbreaking attacks on GitHub Copilot and a 46.3% success rate on Amazon Q. Furthermore, We successfully extracted sensitive user data from GitHub Copilot, including 54 real email addresses and 314 physical addresses associated with GitHub usernames. Our study also demonstrates that these code-based attack methods are effective against general-purpose LLMs, such as the GPT series, highlighting a broader security misalignment in the handling of code by modern LLMs. These findings underscore critical security challenges associated with LCCTs and suggest essential directions for strengthening their security frameworks. The example code and attack samples from our research are provided at https://github.com/Sensente/Security-Attacks-on-LCCTs.