mlsanityData to 14 Oct 2026

GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher

arXiv 2308.06463 ↗First posted 12 Aug 2023cs.CLcited by 423

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Safety lies at the core of the development of Large Language Models (LLMs). There is ample work on aligning LLMs with human ethics and preferences, including data filtering in pretraining, supervised fine-tuning, reinforcement learning from human feedback, and red teaming, etc. In this study, we discover that chat in cipher can bypass the safety alignment techniques of LLMs, which are mainly conducted in natural languages. We propose a novel framework CipherChat to systematically examine the generalizability of safety alignment to non-natural languages -- ciphers. CipherChat enables humans to chat with LLMs through cipher prompts topped with system role descriptions and few-shot enciphered demonstrations. We use CipherChat to assess state-of-the-art LLMs, including ChatGPT and GPT-4 for different representative human ciphers across 11 safety domains in both English and Chinese. Experimental results show that certain ciphers succeed almost 100% of the time to bypass the safety alignment of GPT-4 in several safety domains, demonstrating the necessity of developing safety alignment for non-natural languages. Notably, we identify that LLMs seem to have a ''secret cipher'', and propose a novel SelfCipher that uses only role play and several demonstrations in natural language to evoke this capability. SelfCipher surprisingly outperforms existing human ciphers in almost all cases. Our code and data will be released at https://github.com/RobustNLP/CipherChat.

Key facts. Cited by 423 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 69 references that are in the map. First posted on arXiv 12 Aug 2023.

Builds on

42 of 69 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 40

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2017Deep reinforcement learning from human preferencescited by 3,912
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
28 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  8. 2023LIMA: Less Is More for Alignmentcited by 1,100
  9. 2022Red Teaming Language Models with Language Modelscited by 1,026
  10. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  11. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  13. 2018AI safety via debatecited by 400
  14. 2023Larger language models do in-context learning differentlycited by 394
  15. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  16. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  17. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  18. 2023How is ChatGPT's behavior changing over time?cited by 270
  19. 2023Pretraining Language Models with Human Preferencescited by 247
  20. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  21. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 191
  22. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  23. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 176
  24. 2020Recipes for Safety in Open-domain Chatbotscited by 170
  25. 2021Challenges in Detoxifying Language Modelscited by 169
  26. 2018Supervising strong learners by amplifying weak expertscited by 145
  27. 2023Safety Assessment of Chinese Large Language Modelscited by 85
  28. 2022Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Modelscited by 68

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 21; 2024: 143; 2025: 165; 2026: 94.

423 papers.

Sort:

By area:

  1. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  2. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  4. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  5. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  6. 2023Detecting Language Model Attacks with Perplexitycited by 431
  7. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  8. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  9. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  10. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  11. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
  12. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
38 more
  1. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  2. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  3. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  4. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  5. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 199
  6. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 192
  7. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  8. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 172
  9. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 170
  10. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 153
  11. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  12. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  13. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  14. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 125
  15. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 125
  16. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  17. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  18. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 111
  19. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 110
  20. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  21. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 107
  22. 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Modelscited by 100
  23. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 99
  24. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 86
  25. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  26. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinkingcited by 81
  27. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 80
  28. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 80
  29. 2024Safeguarding Large Language Models: A Surveycited by 78
  30. 2024CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Modelscited by 76
  31. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 76
  32. 2024Intention Analysis Makes LLMs A Good Jailbreak Defendercited by 74
  33. 2024Defending LLMs against Jailbreaking Attacks via Backtranslationcited by 70
  34. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 70
  35. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 69
  36. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 68
  37. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environmentscited by 67
  38. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67

50 most cited of 423 All 423 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Security Concerns for Large Language Models: A Survey
  2. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Security
  3. 2024Diversity Helps Jailbreak Large Language Models
  4. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!
  5. 2024Probing the Safety Response Boundary of Large Language Models via Unsafe Decoding Path Generation
  6. 2024ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates
  7. 2024Root Defence Strategies: Ensuring Safety of LLM at the Decoding Level
  8. 2025An LLM Framework For Cryptography Over Chat Channels