Data to 14 Oct 2026

CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion

arXiv 2403.07865 ↗First posted 12 Mar 2024cs.CL cs.AI cs.CRcited by 76

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

The rapid advancement of Large Language Models (LLMs) has brought about remarkable generative capabilities but also raised concerns about their potential misuse. While strategies like supervised fine-tuning and reinforcement learning from human feedback have enhanced their safety, these methods primarily focus on natural languages, which may not generalize to other domains. This paper introduces CodeAttack, a framework that transforms natural language inputs into code inputs, presenting a novel environment for testing the safety generalization of LLMs. Our comprehensive studies on state-of-the-art LLMs including GPT-4, Claude-2, and Llama-2 series reveal a new and universal safety vulnerability of these models against code input: CodeAttack bypasses the safety guardrails of all models more than 80% of the time. We find that a larger distribution gap between CodeAttack and natural language leads to weaker safety generalization, such as encoding natural language input with data structures. Furthermore, we give our hypotheses about the success of CodeAttack: the misaligned bias acquired by LLMs during code training, prioritizing code completion over avoiding the potential safety risk. Finally, we analyze potential mitigation measures. These findings highlight new safety risks in the code domain and the need for more robust safety alignment algorithms to match the code capabilities of LLMs.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 31 of its 44 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

31 of 44 references.

Closest ancestors · 1

Same direction, newest first

2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 30

2022Training language models to follow instructions with human feedbackcited by 15,971
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
2017Deep reinforcement learning from human preferencescited by 3,912
2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
2023Code Llama: Open Foundation Models for Codecited by 2,139
2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
2022Red Teaming Language Models with Language Modelscited by 1,026
2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651

18 more

2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
2022The Stack: 3 TB of permissively licensed source codecited by 359
2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
2023Pretraining Language Models with Human Preferencescited by 247
2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 191
2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
2023FLIRT: Feedback Loop In-context Red Teamingcited by 83
2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safetycited by 80
2024Identifying Semantic Induction Heads to Understand In-Context Learningcited by 47

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 39; 2026: 30.

76 papers.

Sort:

By area:

2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 69
2025Foot-In-The-Door: A Multi-turn Jailbreak for LLMscited by 54
2024Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understandingcited by 34
2024The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Modelscited by 20
2025QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Languagecited by 18
2025Lifelong Safety Alignment for Language Modelscited by 15
2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
2024Black-Box Adversarial Attacks on LLM-Based Code Completioncited by 14
2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 13
2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methodscited by 12
2025Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Modelscited by 10
2025RedCoder: Automated Multi-Turn Red Teaming for Code LLMscited by 9

38 more

2024POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimizationcited by 8
2025Why Not Act on What You Know? Unleashing Safety Potential of LLMs via Self-Aware Guard Enhancementcited by 8
2025ARMOR: Aligning Secure and Safe Large Language Models via Meticulous Reasoningcited by 8
2025Attention Eclipse: Manipulating Attention to Bypass LLM Safety-Alignmentcited by 7
2025OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalitiescited by 7
2025Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attackscited by 7
2026AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completioncited by 7
2025HumorReject: Decoupling LLM Safety from Refusal Prefix via A Little Humorcited by 6
2025Bleeding Pathways: Vanishing Discriminability in LLM Hidden States Fuels Jailbreak Attackscited by 5
2025Smoke and Mirrors: Jailbreaking LLM-based Code Generation via Implicit Malicious Promptscited by 5
2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 5
2024FATS: A Prompt Injection Attack Utilizing Feign Security Agents with Deceptive Few-shots Learningcited by 4
2024SQL Injection Jailbreak: A Structural Disaster of Large Language Modelscited by 4
2025When Grammar Guides the Attack: Uncovering Control-Plane Vulnerabilities in LLMs with Structured Outputcited by 4
2025Imperceptible Jailbreaking against Large Language Modelscited by 4
2025Characterizing Selective Refusal Bias in Large Language Modelscited by 4
2026Defending Large Language Models Against Jailbreak Attacks via In-Decoding Safety-Awareness Probingcited by 4
2026What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistantscited by 4
2025HarmRLVR: Weaponizing Verifiable Rewards for Harmful LLM Alignmentcited by 3
2026STAR-S: Improving Safety Alignment through Self-Taught Reasoning on Safety Rulescited by 3
2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 3
2025StructTransform: A Scalable Attack Surface for Safety-Aligned Large Language Modelscited by 2
2025InfoFlood: Jailbreaking Large Language Models with Information Overloadcited by 2
2025ERIS: Evolutionary Real-world Interference Scheme for Jailbreaking Audio Large Modelscited by 2
2025Jailbreaking LLMs via Semantically Relevant Nested Scenarios with Targeted Toxic Knowledgecited by 2
2025ArtPerception: ASCII Art-based Jailbreak on LLMs with Recognition Pre-testcited by 2
2026CASA: Classification Augmented with Safety Attention for Robust Multimodal Alignmentcited by 2
2026Involuntary In-Context Learning: Exploiting Few-Shot Pattern Completion to Bypass Safety Alignment in GPT-5.4cited by 2
2026Furina: Fragmented Uncertainty-Driven Refusal Instability Attackcited by 2
2026Structuring the Space of Sociotechnical Alignmentcited by 2
2025Self-Instruct Few-Shot Jailbreaking: Decompose the Attack into Pattern and Behavior Learningcited by 1
2025GraphAttack: Exploiting Representational Blindspots in LLM Safety Mechanismscited by 1
2025Activation-Guided Local Editing for Jailbreaking Attackscited by 1
2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
2025SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarkingcited by 1
2025ASTRA: An Automated Framework for Strategy Discovery, Retrieval, and Evolution for Jailbreaking LLMscited by 1
2025Why Does Weak-OOD Help? A Further Step Towards Understanding Jailbreaking VLMscited by 1
2025Understanding and Mitigating Over-refusal for Large Language Models via Representation Interventioncited by 1

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way