mlsanityData to 7 Oct 2026

Voice Jailbreak Attacks Against GPT-4o

arXiv 2405.19103 ↗First posted 29 May 2024cs.CR cs.LGcited by 39

On the map: LLM safety and security → Audio jailbreaks and speech backdoors

Abstract (arXiv)

Recently, the concept of artificial assistants has evolved from science fiction into real-world applications. GPT-4o, the newest multimodal large language model (MLLM) across audio, vision, and text, has further blurred the line between fiction and reality by enabling more natural human-computer interactions. However, the advent of GPT-4o's voice mode may also introduce a new attack surface. In this paper, we present the first systematic measurement of jailbreak attacks against the voice mode of GPT-4o. We show that GPT-4o demonstrates good resistance to forbidden questions and text jailbreak prompts when directly transferring them to voice mode. This resistance is primarily due to GPT-4o's internal safeguards and the difficulty of adapting text jailbreak prompts to voice mode. Inspired by GPT-4o's human-like behaviors, we propose VoiceJailbreak, a novel voice jailbreak attack that humanizes GPT-4o and attempts to persuade it through fictional storytelling (setting, character, and plot). VoiceJailbreak is capable of generating simple, audible, yet effective jailbreak prompts, which significantly increases the average attack success rate (ASR) from 0.033 to 0.778 in six forbidden scenarios. We also conduct extensive experiments to explore the impacts of interaction steps, key elements of fictional writing, and different languages on VoiceJailbreak's effectiveness and further enhance the attack performance with advanced fictional writing techniques. We hope our study can assist the research community in building more secure and well-regulated MLLMs.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Audio jailbreaks and speech backdoors, one of 12 directions in LLM safety and security. Builds on 28 of its 44 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

28 of 44 references.

References · 28

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2020Extracting Training Data from Large Language Modelscited by 2,144
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  8. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  9. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  10. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  11. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  12. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
16 more
  1. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  2. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  3. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  4. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  5. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  6. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  7. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  8. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  9. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  10. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  11. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  12. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  13. 2023Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Modelscited by 171
  14. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  15. 2023A Pathway Towards Responsible AI Generated Contentcited by 38
  16. 2024Malla: Demystifying Real-world Large Language Model Integrated Malicious Servicescited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 21; 2026: 14.

39 papers.

Sort:

By area:

  1. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  2. 2024Best-of-N Jailbreakingcited by 71
  3. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  4. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  5. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 32
  6. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  7. 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 25
  8. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 16
  9. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 15
  10. 2025Survey of Adversarial Robustness in Multimodal Large Language Modelscited by 14
  11. 2025JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Modelscited by 11
  12. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
27 more
  1. 2025Aligning Spoken Dialogue Models from User Interactionscited by 11
  2. 2025Invisible Prompts, Visible Threats: Malicious Font Injection in External Resources for Large Language Modelscited by 9
  3. 2025"I am bad": Interpreting Stealthy, Universal and Robust Audio Jailbreaks in Audio-Language Modelscited by 7
  4. 2025`From Prompt to Perturbation': An Adaptive Framework for Voice-Based Jailbreaks on Audio LLMscited by 6
  5. 2026Now You Hear Me: Audio Narrative Attacks Against Large Audio-Language Modelscited by 5
  6. 2025Audio Jailbreak Attacks: Exposing Vulnerabilities in SpeechGPT in a White-Box Frameworkcited by 4
  7. 2025Chain-of-Code Collapse: Reasoning Failures in LLMs via Adversarial Prompting in Code Generationcited by 4
  8. 2025Breaking Audio Large Language Models by Attacking Only the Encoder: A Universal Targeted Latent-Space Audio Attackcited by 3
  9. 2026Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injectioncited by 2
  10. 2024Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Modelscited by 1
  11. 2025Investigating Vulnerabilities and Defenses Against Audio-Visual Attacks: A Comprehensive Survey Emphasizing Multimodal Modelscited by 1
  12. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  13. 2025Rebellion: Noise-Robust Reasoning Training for Audio Reasoning Modelscited by 1
  14. 2026Detecting Mental Manipulation in Speech via Synthetic Multi-Speaker Dialoguecited by 1
  15. 2026The Salami Slicing Threat: Exploiting Cumulative Risks in LLM Systemscited by 1
  16. 2026Benign Fine-Tuning Breaks Safety Alignment in Audio LLMscited by 1
  17. 2026Voices Across Registers: Corpus-Conditioned Vernacular Jailbreaks against Aligned LLMs via Fanfiction Subgenrescited by 1
  18. 2025Adversarial Attacks in Multimodal Systems: A Practitioner's Surveycited by 0
  19. 2025Multilingual Collaborative Defense for Large Language Modelscited by 0
  20. 2026The Alignment Curse: Modality Alignment Supercharges Audio Attacks via Text Transfercited by 0
  21. 2026Aegis: Towards Governance, Integrity, and Security of AI Voice Agentscited by 0
  22. 2026Evaluation of Audio Language Models for Fairness, Safety, and Securitycited by 0
  23. 2026Codec-Robust Attacks on Audio LLMscited by 0
  24. 2026SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speechcited by 0
  25. 2026Prosody-driven Jailbreaks in Audio LLMs: A Controlled Study and Mechanistic Analysiscited by 0
  26. 2026Piggybacking on Perception: Stealthy Concurrent Audio Prompt Injections against Multimodal LLM Agentscited by 0
  27. 2026Psychoacoustically Aligned Latent Smoothing for Adversarial Robustness of Full-Duplex Speech-to-Speech Dialogue Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts
  2. 2024Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?
  3. 2024Can Large Language Models Automatically Jailbreak GPT-4V?
  4. 2025Jailbreak Mimicry: Automated Discovery of Narrative-Based Jailbreaks for Large Language Models
  5. 2025Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Models
  6. 2024MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialogue
  7. 2024Jailbreaking Attack against Multimodal Large Language Model
  8. 2025The Tower of Babel Revisited: Multilingual Jailbreak Prompts on Closed-Source Large Language Models