mlsanityData to 7 Oct 2026

CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models

arXiv 2402.16717 ↗First posted 26 Feb 2024cs.CL cs.AI cs.CRcited by 65

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Adversarial misuse, particularly through `jailbreaking' that circumvents a model's safety and ethical protocols, poses a significant challenge for Large Language Models (LLMs). This paper delves into the mechanisms behind such successful attacks, introducing a hypothesis for the safety mechanism of aligned LLMs: intent security recognition followed by response generation. Grounded in this hypothesis, we propose CodeChameleon, a novel jailbreak framework based on personalized encryption tactics. To elude the intent security recognition phase, we reformulate tasks into a code completion format, enabling users to encrypt queries using personalized encryption functions. To guarantee response generation functionality, we embed a decryption function within the instructions, which allows the LLM to decrypt and execute the encrypted queries successfully. We conduct extensive experiments on 7 LLMs, achieving state-of-the-art average Attack Success Rate (ASR). Remarkably, our method achieves an 86.6% ASR on GPT-4-1106.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 21 of its 30 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

21 of 30 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  2. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  3. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  5. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  6. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  6. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  8. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  9. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  10. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  11. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  12. 2023CodeIE: Large Code Generation Models are Better Few-Shot Information Extractorscited by 67
3 more
  1. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  2. 2023On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?cited by 25
  3. 2023Prompting with Pseudo-Code Instructionscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 34; 2026: 17.

65 papers.

Sort:

By area:

  1. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  2. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 185
  3. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  4. 2024Safeguarding Large Language Models: A Surveycited by 73
  5. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  6. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  7. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  8. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  9. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 32
  10. 2024PyRIT: A Framework for Security Risk Identification and Red Teaming in Generative AI Systemcited by 29
  11. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  12. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
38 more
  1. 2024The Better Angels of Machine Personality: How Personality Relates to LLM Safetycited by 21
  2. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  3. 2024The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Modelscited by 20
  4. 2024PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMscited by 17
  5. 2024You Know What I'm Saying: Jailbreak Attack via Implicit Referencecited by 13
  6. 2025One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMscited by 12
  7. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  8. 2025Think Twice, Generate Once: Safeguarding by Progressive Self-Reflectioncited by 11
  9. 2025Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcarecited by 10
  10. 2025Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMscited by 10
  11. 2026SEMA: Simple yet Effective Learning for Multi-Turn Jailbreak Attackscited by 10
  12. 2024PathSeeker: Exploring LLM Security Vulnerabilities with a Reinforcement Learning-Based Jailbreak Approachcited by 9
  13. 2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methodscited by 9
  14. 2025Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Modelscited by 9
  15. 2025SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinationscited by 9
  16. 2024Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attackscited by 8
  17. 2025PiCo: Jailbreaking Multimodal Large Language Models via Pictorial Code Contextualizationcited by 8
  18. 2025Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debatecited by 8
  19. 2025ASTRA: Autonomous Spatial-Temporal Red-teaming for AI Software Assistantscited by 6
  20. 2025Bypassing Prompt Guards in Production with Controlled-Release Promptingcited by 6
  21. 2025HumorReject: Decoupling LLM Safety from Refusal Prefix via A Little Humorcited by 5
  22. 2025Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Modelscited by 5
  23. 2025KDA: A Knowledge-Distilled Attacker for Generating Diverse Prompts to Jailbreak LLMscited by 4
  24. 2025Chain-of-Thought Driven Adversarial Scenario Extrapolation for Robust Language Modelscited by 4
  25. 2025Evo-MARL: Co-Evolutionary Multi-Agent Reinforcement Learning for Internalized Safetycited by 4
  26. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  27. 2025StructTransform: A Scalable Attack Surface for Safety-Aligned Large Language Modelscited by 2
  28. 2025Effective and Efficient Jailbreaks of Black-Box LLMs with Cross-Behavior Attackscited by 2
  29. 2025AdaptiveGuard: Towards Adaptive Runtime Safety for LLM-Powered Softwarecited by 2
  30. 2025EquaCode: A Multi-Strategy Jailbreak Approach for Large Language Models via Equation Solving and Code Completioncited by 2
  31. 2026How Real is Your Jailbreak? Fine-grained Jailbreak Evaluation with Anchored Referencecited by 2
  32. 2026Stay in Character, Stay Safe: Dual-Cycle Adversarial Self-Evolution for Safety Role-Playing Agentscited by 2
  33. 2026Invisible Safety Threat: Malicious Finetuning for LLM via Steganographycited by 2
  34. 2026RouteScan: A Non-Intrusive Approach to Auditing MoE LLMs Safety via Expert Routing Telemetrycited by 2
  35. 2025Beyond Surface-Level Detection: Towards Cognitive-Driven Defense Against Jailbreak Attacks via Meta-Operations Reasoningcited by 1
  36. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  37. 2025AutoRed: A Free-form Adversarial Prompt Generation Framework for Automated Red Teamingcited by 1
  38. 2025Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Modelscited by 1

50 most cited of 65 All 65 (JSON)