mlsanityData to 7 Oct 2026

Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts

arXiv 2311.09127 ↗First posted 15 Nov 2023cs.CR cs.AI cs.LGcited by 62

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Existing work on jailbreak Multimodal Large Language Models (MLLMs) has focused primarily on adversarial examples in model inputs, with less attention to vulnerabilities, especially in model API. To fill the research gap, we carry out the following work: 1) We discover a system prompt leakage vulnerability in GPT-4V. Through carefully designed dialogue, we successfully extract the internal system prompts of GPT-4V. This finding indicates potential exploitable security risks in MLLMs; 2) Based on the acquired system prompts, we propose a novel MLLM jailbreaking attack method termed SASP (Self-Adversarial Attack via System Prompt). By employing GPT-4 as a red teaming tool against itself, we aim to search for potential jailbreak prompts leveraging stolen system prompts. Furthermore, in pursuit of better performance, we also add human modification based on GPT-4's analysis, which further improves the attack success rate to 98.7%; 3) We evaluated the effect of modifying system prompts to defend against jailbreaking attacks. Results show that appropriately designed system prompts can significantly reduce jailbreak success rates. Overall, our work provides new insights into enhancing MLLM security, demonstrating the important role of system prompts in jailbreaking. This finding could be leveraged to greatly facilitate jailbreak success rates while also holding the potential for defending against jailbreaks.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 19 of its 27 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

19 of 27 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  3. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots16 Jul
  4. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  5. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  7. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  8. 2023Pretraining Language Models with Human Preferencescited by 245
  9. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  10. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  11. 2023Effective Prompt Extraction from Language Modelscited by 83
  12. 2023DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agentscited by 52
2 more
  1. 2023The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Taskscited by 35
  2. 2023Promptor: A Conversational and Autonomous Prompt Generation Agent for Intelligent Text Entry Techniquescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 36; 2025: 22; 2026: 2.

62 papers.

Sort:

By area:

  1. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  2. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  3. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Modelscited by 121
  4. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 114
  5. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 95
  6. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 82
  7. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  8. 2024Safety of Multimodal Large Language Models on Images and Textscited by 70
  9. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  10. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safetycited by 59
  11. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  12. 2024Machine Unlearning in Generative AI: A Surveycited by 49
38 more
  1. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  2. 2024AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Modelscited by 43
  3. 2024Core Knowledge Deficits in Multi-Modal Language Modelscited by 38
  4. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  5. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  6. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  7. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  8. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  9. 2025Distraction is All You Need for Multimodal Large Language Model Jailbreakingcited by 31
  10. 2025DoomArena: A framework for Testing AI Agents Against Evolving Security Threatscited by 31
  11. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden Statescited by 29
  12. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  13. 2024Robust Concept Erasure Using Task Vectorscited by 27
  14. 2024Ranking Manipulation for Conversational Search Enginescited by 23
  15. 2024Adversarial Attacks of Vision Tasks in the Past 10 Years: A Surveycited by 22
  16. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defensecited by 20
  17. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 18
  18. 2025The Pitfalls of KV Cache Compressioncited by 15
  19. 2024Energy-Latency Manipulation of Multi-modal Large Language Models via Verbose Samplescited by 14
  20. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  21. 2025Survey of Adversarial Robustness in Multimodal Large Language Modelscited by 14
  22. 2025Attack and defense techniques in large language models: A survey and new perspectivescited by 13
  23. 2025One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMscited by 12
  24. 2024Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Modelscited by 11
  25. 2024LLMs Can Defend Themselves Against Jailbreaking in a Practical Manner: A Vision Papercited by 9
  26. 2024Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Modelscited by 9
  27. 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Modelscited by 9
  28. 2025Better Language Model Inversion by Compactly Representing Next-Token Distributionscited by 9
  29. 2024Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attackscited by 8
  30. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  31. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
  32. 2024DORY: Deliberative Prompt Recovery for LLMcited by 7
  33. 2025Spot Risks Before Speaking! Unraveling Safety Attention Heads in Large Vision-Language Modelscited by 7
  34. 2024Human-Interpretable Adversarial Prompt Attack on Large Language Models with Situational Contextcited by 6
  35. 2025Layer-Level Self-Exposure and Patch: Affirmative Token Mitigation for Jailbreak Attack Defensecited by 6
  36. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  37. 2025Making Every Step Effective: Jailbreaking Large Vision-Language Models Through Hierarchical KV Equalizationcited by 4
  38. 2025Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag Paradigmcited by 4

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?
  2. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks
  3. 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
  4. 2025Jailbreak Mimicry: Automated Discovery of Narrative-Based Jailbreaks for Large Language Models
  5. 2024Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translation
  6. 2025The Tower of Babel Revisited: Multilingual Jailbreak Prompts on Closed-Source Large Language Models
  7. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking
  8. 2024Jailbreaking Attack against Multimodal Large Language Model