# Minimal Prompt Perturbations Lead to Code Vulnerabilities: Prompt Fragility and Hidden-State Signals in Coding LLMs

arXiv: https://arxiv.org/abs/2605.29737 · HTML: https://mlsanity.com/paper/2605.29737/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 17 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 13 of 13)

- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [ReCode: Robustness Evaluation of Code Generation Models](https://mlsanity.com/paper/2212.10264/) (2022)
- [CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation](https://mlsanity.com/paper/2501.08200/) (2025)
- [NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations](https://mlsanity.com/paper/2406.19783/) (2024)
- [Who is using AI to code? Global diffusion and impact of generative AI](https://mlsanity.com/paper/2506.08945/) (2025)
- [Correctness Assessment of Code Generated by Large Language Models Using Internal Representations](https://mlsanity.com/paper/2501.12934/) (2025)
- [Exploring the Robustness of Large Language Models for Solving Programming Problems](https://mlsanity.com/paper/2306.14583/) (2023)
- [A Multi-Language Perspective on the Robustness of LLM Code Generation](https://mlsanity.com/paper/2504.19108/) (2025)
- [CREME: Robustness Enhancement of Code LLMs via Layer-Aware Model Editing](https://mlsanity.com/paper/2507.16407/) (2025)
- [How Secure is Secure Code Generation? Adversarial Prompts Put LLM Defenses to the Test](https://mlsanity.com/paper/2601.07084/) (2026)

## Built on by (most cited first, 2 of 2)

- [Probing the Prefill: Detecting Code Vulnerabilities via Latent Activations](https://mlsanity.com/paper/2608.16970/) (2026)
- [ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time](https://mlsanity.com/paper/2610.02363/) (2026)

## Abstract (arXiv)

LLM-based coding assistants are seeing rapid adoption, offering substantial gains in developer productivity. As organizations increasingly ship code these agents produce, the security of that code becomes critical. Prior work has shown that minor prompt perturbations degrade the functional correctness of LLM-generated code, but whether they also compromise code security has remained unstudied. We apply token-level mutations to prompts across three models and five programming languages, and show that mutations as small as a single-character change can flip generated code from secure to vulnerable. Probing the models' hidden states reveals that this fragility is partially encoded in prompt representations, but unevenly so. Input-handling vulnerabilities, where the model omits validation or sanitization, are more predictable (mean AUC 0.753) than secure-defaults vulnerabilities, where insecure code stems from one local choice such as a weak algorithm or unsafe parameter (mean AUC 0.674). These results show that the threat model for LLM-assisted coding extends beyond prompt injection to ordinary prompt variation, and indicate that input-handling flaws can be caught before generation while secure-defaults flaws require intervention during decoding.