mlsanityData to 7 Oct 2026

Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attacks

arXiv 2406.06302 ↗First posted 10 Jun 2024cs.CR cs.CVcited by 35

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

The recent release of GPT-4o has garnered widespread attention due to its powerful general capabilities. While its impressive performance is widely acknowledged, its safety aspects have not been sufficiently explored. Given the potential societal impact of risky content generated by advanced generative AI such as GPT-4o, it is crucial to rigorously evaluate its safety. In response to this question, this paper for the first time conducts a rigorous evaluation of GPT-4o against jailbreak attacks. Specifically, this paper adopts a series of multi-modal and uni-modal jailbreak attacks on 4 commonly used benchmarks encompassing three modalities (ie, text, speech, and image), which involves the optimization of over 4,000 initial text queries and the analysis and statistical evaluation of nearly 8,000+ response on GPT-4o. Our extensive experiments reveal several novel observations: (1) In contrast to the previous version (such as GPT-4V), GPT-4o has enhanced safety in the context of text modality jailbreak; (2) The newly introduced audio modality opens up new attack vectors for jailbreak attacks on GPT-4o; (3) Existing black-box multimodal jailbreak attack methods are largely ineffective against GPT-4o and GPT-4V. These findings provide critical insights into the safety implications of GPT-4o and underscore the need for robust alignment guardrails in large models. Our code is available at urlhttps://github.com/NY1024/JailbreakGPT4o.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 23 of its 36 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

23 of 36 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  2. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  5. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  8. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  9. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  10. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  11. 2021Dual Attention Suppression Attack: Generate Adversarial Camouflage in Physical Worldcited by 127
  12. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
6 more
  1. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Modelscited by 83
  2. 2023Safety Assessment of Chinese Large Language Modelscited by 82
  3. 2023X-Adv: Physical Adversarial Object Attacks against X-ray Prohibited Item Detectioncited by 58
  4. 2023Pre-trained Trojan Attacks for Visual Recognitioncited by 38
  5. 2024Poisoned Forgery Face: Towards Backdoor Attacks on Face Forgery Detectioncited by 34
  6. 2023Does Few-shot Learning Suffer from Backdoor Attacks?cited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 27; 2026: 4.

35 papers.

Sort:

By area:

  1. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 82
  2. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  3. 2025Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Modelscited by 61
  4. 2025Jailbreaking Multimodal Large Language Models via Shuffle Inconsistencycited by 42
  5. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  6. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructionscited by 33
  7. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 32
  8. 2024Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiencycited by 31
  9. 2025ThinkGuard: Deliberative Slow Thinking Leads to Cautious Guardrailscited by 29
  10. 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 25
  11. 2025Towards Understanding the Safety Boundaries of DeepSeek Models: Evaluation and Findingscited by 24
  12. 2024Adversarial Attacks of Vision Tasks in the Past 10 Years: A Surveycited by 22
23 more
  1. 2025SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agentscited by 17
  2. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 16
  3. 2025Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sinkcited by 15
  4. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 15
  5. 2025Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025cited by 11
  6. 2025`From Prompt to Perturbation': An Adaptive Framework for Voice-Based Jailbreaks on Audio LLMscited by 6
  7. 2025POISONCRAFT: Practical Poisoning of Retrieval-Augmented Generation for Large Language Modelscited by 6
  8. 2026Now You Hear Me: Audio Narrative Attacks Against Large Audio-Language Modelscited by 5
  9. 2025REVEAL: Multi-turn Evaluation of Image-Input Harms for Vision LLMcited by 4
  10. 2025Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variationscited by 4
  11. 2025SPARK: Jailbreaking T2V Models by Synergistically Prompting Auditory and Recontextualized Knowledgecited by 4
  12. 2025PRISM: Programmatic Reasoning with Image Sequence Manipulation for LVLM Jailbreakingcited by 3
  13. 2026Jailbreaking Vision-Language Models Through the Visual Modalitycited by 3
  14. 2025The Illusionist's Prompt: Exposing the Factual Vulnerabilities of Large Language Models with Linguistic Nuancescited by 2
  15. 2025PRJ: Perception-Retrieval-Judgement for Generated Imagescited by 2
  16. 2025Sequential Comics for Jailbreaking Multimodal Large Language Models via Structured Visual Storytellingcited by 2
  17. 2026Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injectioncited by 2
  18. 2025Investigating Vulnerabilities and Defenses Against Audio-Visual Attacks: A Comprehensive Survey Emphasizing Multimodal Modelscited by 1
  19. 2025Animating Language Practice: Engagement with Stylized Conversational Agents in Japanese Learningcited by 1
  20. 2025VisualDAN: Exposing Vulnerabilities in VLMs with Visual-Driven DAN Commandscited by 1
  21. 2025Rebellion: Noise-Robust Reasoning Training for Audio Reasoning Modelscited by 1
  22. 2026Evolving Deception: When Agents Evolve, Deception Winscited by 1
  23. 2024CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Modelcited by 0