mlsanityData to 7 Oct 2026

Image-to-Text Logic Jailbreak: Your Imagination can Help You Do Anything

arXiv 2407.02534 ↗First posted 1 Jul 2024cs.CR cs.CVcited by 7

On the map: LLM safety and security → Vision-language and text-to-image safety

Abstract (arXiv)

Large Visual Language Modeltextbfs (VLMs) such as GPT-4V have achieved remarkable success in generating comprehensive and nuanced responses. Researchers have proposed various benchmarks for evaluating the capabilities of VLMs. With the integration of visual and text inputs in VLMs, new security issues emerge, as malicious attackers can exploit multiple modalities to achieve their objectives. This has led to increasing attention on the vulnerabilities of VLMs to jailbreak. Most existing research focuses on generating adversarial images or nonsensical image to jailbreak these models. However, no researchers evaluate whether logic understanding capabilities of VLMs in flowchart can influence jailbreak. Therefore, to fill this gap, this paper first introduces a novel dataset Flow-JD specifically designed to evaluate the logic-based flowchart jailbreak capabilities of VLMs. We conduct an extensive evaluation on GPT-4o, GPT-4V, other 5 SOTA open source VLMs and the jailbreak rate is up to 92.8%. Our research reveals significant vulnerabilities in current VLMs concerning image-to-text jailbreak and these findings underscore the the urgency for the development of robust and effective future defenses.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security. Builds on 26 of its 34 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

26 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Automatic Jailbreaking of the Text-to-Image Generative AI Systems26 May

Other references · 25

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  4. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  9. 2023Large Language Models as Optimizerscited by 830
  10. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  11. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  12. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
13 more
  1. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  2. 2023Are aligned neural networks adversarially aligned?cited by 305
  3. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  4. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  5. 2021A Survey On Universal Adversarial Attackcited by 59
  6. 2023Fairness-guided Few-shot Prompting for Large Language Modelscited by 53
  7. 2021Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!cited by 51
  8. 2021Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Modelscited by 29
  9. 2024How Far Are We from Intelligent Visual Deductive Reasoning?cited by 29
  10. 2024Bridge the Modality and Capability Gaps in Vision-Language Model Selectioncited by 20
  11. 2024Can LLMs' Tuning Methods Work in Medical Multimodal Domain?cited by 12
  12. 2023Expanding Scope: Adapting English Adversarial Attacks to Chinesecited by 3
  13. 2024Discovering Universal Semantic Triggers for Text-to-Image Synthesiscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  2. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  3. 2025JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steeringcited by 10
  4. 2025On Surjectivity of Neural Networks: Can you elicit any behavior from your model?cited by 4
  5. 2025Agentic Moderation: Multi-Agent Design for Safer Vision-Language Modelscited by 1
  6. 2026Hierarchical Attacks for Multi-Modal Multi-Agent Reasoningcited by 1
  7. 2026SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift
  2. 2024IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselves
  3. 2026MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models
  4. 2025JaiLIP: Jailbreaking Vision-Language Models via Loss Guided Image Perturbation
  5. 2024Failures to Find Transferable Image Jailbreaks Between Vision-Language Models
  6. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image
  7. 2025VisualDAN: Exposing Vulnerabilities in VLMs with Visual-Driven DAN Commands
  8. 2024Jailbreaking Attack against Multimodal Large Language Model