mlsanityData to 7 Oct 2026

ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs

arXiv 2402.11753 ↗First posted 19 Feb 2024cs.CL cs.AIcited by 197

On the map: LLM safety and security → Vision-language and text-to-image safety

Abstract (arXiv)

Safety is critical to the usage of large language models (LLMs). Multiple techniques such as data filtering and supervised fine-tuning have been developed to strengthen LLM safety. However, currently known techniques presume that corpora used for safety alignment of LLMs are solely interpreted by semantics. This assumption, however, does not hold in real-world applications, which leads to severe vulnerabilities in LLMs. For example, users of forums often use ASCII art, a form of text-based art, to convey image information. In this paper, we propose a novel ASCII art-based jailbreak attack and introduce a comprehensive benchmark Vision-in-Text Challenge (ViTC) to evaluate the capabilities of LLMs in recognizing prompts that cannot be solely interpreted by semantics. We show that five SOTA LLMs (GPT-3.5, GPT-4, Gemini, Claude, and Llama2) struggle to recognize prompts provided in the form of ASCII art. Based on this observation, we develop the jailbreak attack ArtPrompt, which leverages the poor performance of LLMs in recognizing ASCII art to bypass safety measures and elicit undesired behaviors from LLMs. ArtPrompt only requires black-box access to the victim LLMs, making it a practical attack. We evaluate ArtPrompt on five SOTA LLMs, and show that ArtPrompt can effectively and efficiently induce undesired behaviors from all five LLMs. Our code is available at https://github.com/uw-nsl/ArtPrompt.

Key facts. Cited by 197 papers in the map (data through 7 Oct 2026). Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security. Builds on 37 of its 47 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

37 of 47 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2020Extracting Training Data from Large Language Modelscited by 2,144
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
25 more
  1. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  3. 2021Ethical and social risks of harm from Language Modelscited by 957
  4. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  6. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  7. 2019Release Strategies and the Social Impacts of Language Modelscited by 505
  8. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  9. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  10. 2023Detecting Language Model Attacks with Perplexitycited by 411
  11. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  12. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  13. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  14. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  15. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  16. 2022Predictability and Surprise in Large Generative Modelscited by 209
  17. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  18. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  19. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  20. 2023Planning with Large Language Models for Code Generationcited by 179
  21. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  22. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  23. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  24. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  25. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 43; 2025: 98; 2026: 56.

197 papers.

Sort:

By area:

  1. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  2. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  3. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 185
  4. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  5. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  6. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 133
  7. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 130
  8. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  9. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  10. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  11. 2024Imprompter: Tricking LLM Agents into Improper Tool Usecited by 66
  12. 2024Steering Without Side Effects: Improving Post-Deployment Control of Language Modelscited by 44
38 more
  1. 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agentcited by 39
  2. 2025A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaoscited by 39
  3. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 38
  4. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  5. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 32
  6. 2024ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templatescited by 29
  7. 2024WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Responsecited by 28
  8. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  9. 2024MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialoguecited by 28
  10. 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 25
  11. 2024Ranking Manipulation for Conversational Search Enginescited by 23
  12. 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Modelscited by 23
  13. 2025Steering MoE LLMs via Expert (De)Activationcited by 23
  14. 2025Lessons From Red Teaming 100 Generative AI Productscited by 22
  15. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 21
  16. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
  17. 2024Endless Jailbreaks with Bijection Learningcited by 18
  18. 2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 18
  19. 2024Large Language Models Are Involuntary Truth-Tellers: Exploiting Fallacy Failure for Jailbreak Attackscited by 17
  20. 2025PurpCode: Reasoning for Safer Code Generationcited by 17
  21. 2025QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Languagecited by 16
  22. 2024Whiteboard-of-Thought: Thinking Step-by-Step Across Modalitiescited by 15
  23. 2025Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interfacecited by 15
  24. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 15
  25. 2024Jailbreaking Large Language Models with Symbolic Mathematicscited by 14
  26. 2024BELLS: A Framework Towards Future Proof Benchmarks for the Evaluation of LLM Safeguardscited by 13
  27. 2025Lifelong Safety Alignment for Language Modelscited by 13
  28. 2025Adversarial Tokenizationcited by 11
  29. 2025CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challengescited by 11
  30. 2025SoSBench: Benchmarking Safety Alignment on Six Scientific Domainscited by 11
  31. 2025TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewardscited by 11
  32. 2024SATA: A Paradigm for LLM Jailbreak via Simple Assistive Task Linkagecited by 10
  33. 2025Jailbreak Attack Initializations as Extractors of Compliance Directionscited by 10
  34. 2025RainbowPlus: Enhancing Adversarial Prompt Generation via Evolutionary Quality-Diversity Searchcited by 10
  35. 2025MixAT: Combining Continuous and Discrete Adversarial Training for LLMscited by 10
  36. 2025Enhancing Jailbreak Attacks on LLMs via Persona Promptscited by 10
  37. 2026SEMA: Simple yet Effective Learning for Multi-Turn Jailbreak Attackscited by 10
  38. 2026PIArena: A Platform for Prompt Injection Evaluationcited by 10

50 most cited of 197 All 197 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models
  2. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts
  3. 2025PolyJailbreak: Cross-Modal Jailbreaking Attacks on Black-Box Multimodal LLMs
  4. 2025Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models
  5. 2026Text is All You Need for Vision-Language Model Jailbreaking
  6. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image
  7. 2026MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models
  8. 2026Emoji-Based Jailbreaking of Large Language Models