mlsanityData to 7 Oct 2026

When "Competency" in Reasoning Opens the Door to Vulnerability: Jailbreaking LLMs via Novel Complex Ciphers

arXiv 2402.10601 ↗First posted 16 Feb 2024cs.CL cs.AIcited by 27

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Recent advancements in Large Language Model (LLM) safety have primarily focused on mitigating attacks crafted in natural language or common ciphers (e.g. Base64), which are likely integrated into newer models' safety training. However, we reveal a paradoxical vulnerability: as LLMs advance in reasoning, they inadvertently become more susceptible to novel jailbreaking attacks. Enhanced reasoning enables LLMs to interpret complex instructions and decode complex user-defined ciphers, creating an exploitable security gap. To study this vulnerability, we introduce Attacks using Custom Encryptions (ACE), a jailbreaking technique that encodes malicious queries with novel ciphers. Extending ACE, we introduce Layered Attacks using Custom Encryptions (LACE), which applies multi-layer ciphers to amplify attack complexity. Furthermore, we develop CipherBench, a benchmark designed to evaluate LLMs' accuracy in decoding encrypted benign text. Our experiments reveal a critical trade-off: LLMs that are more capable of decoding ciphers are more vulnerable to LACE, with success rates on gpt-oss-20b escalating from 60% under ACE to 72% with LACE. These findings highlight a critical insight: as LLMs become more adept at deciphering complex user ciphers--many of which cannot be preemptively included in safety training--they become increasingly exploitable.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 25 of its 44 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

25 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  11. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  12. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
12 more
  1. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  2. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  3. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  4. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  5. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  6. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  7. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  8. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  9. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  10. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  11. 2023Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Informationcited by 32
  12. 2023Summon a Demon and Bind it: A Grounded Theory of LLM Red Teamingcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 15; 2026: 4.

27 papers.

Sort:

By area:

  1. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  2. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  3. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  4. 2024Defending LLMs against Jailbreaking Attacks via Backtranslationcited by 65
  5. 2024Jailbreak Large Vision-Language Models Through Multi-Modal Linkagecited by 53
  6. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  7. 2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QAcited by 28
  8. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  9. 2024Emerging Vulnerabilities in Frontier Models: Multi-Turn Jailbreak Attackscited by 16
  10. 2024Cognitive Overload Attack:Prompt Injection for Long Contextcited by 13
  11. 2025CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challengescited by 11
  12. 2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methodscited by 9
15 more
  1. 2025Three Minds, One Legend: Jailbreak Large Reasoning Model with Adaptive Stacked Cipherscited by 8
  2. 2025All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Languagecited by 7
  3. 2025Attention Eclipse: Manipulating Attention to Bypass LLM Safety-Alignmentcited by 6
  4. 2025Should LLM Safety Be More Than Refusing Harmful Instructions?cited by 6
  5. 2025ASTRA: Autonomous Spatial-Temporal Red-teaming for AI Software Assistantscited by 6
  6. 2025Benchmarking Large Language Models for Cryptanalysis and Side-Channel Vulnerabilitiescited by 5
  7. 2025Cross-Modal Obfuscation for Jailbreak Attacks on Large Vision-Language Modelscited by 5
  8. 2026AISA: Awakening Intrinsic Safety Awareness in Large Language Models against Jailbreak Attackscited by 3
  9. 2025StructTransform: A Scalable Attack Surface for Safety-Aligned Large Language Modelscited by 2
  10. 2025Jailbreaking Large Language Models Through Content Concretizationcited by 2
  11. 2026Invisible Safety Threat: Malicious Finetuning for LLM via Steganographycited by 2
  12. 2026Intent Laundering: AI Safety Datasets Are Not What They Seemcited by 1
  13. 2025TestDG: Test-time Domain Generalization for Continual Test-time Adaptationcited by 0
  14. 2025OptAgent: Optimizing Query Rewriting for E-commerce via Multi-Agent Simulationcited by 0
  15. 2026RoguePrompt: Dual-Layer Encoding for Self-Reconstruction to Circumvent LLM Moderationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Root Defence Strategies: Ensuring Safety of LLM at the Decoding Level
  2. 2025Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures
  3. 2025Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers
  4. 2026Analysis of LLMs Against Prompt Injection and Jailbreak Attacks
  5. 2025PUZZLED: Jailbreaking LLMs through Word-Based Puzzles
  6. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment
  7. 2026Defending Large Language Models Against Jailbreak Attacks via In-Decoding Safety-Awareness Probing
  8. 2025A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaos