# Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?

arXiv: https://arxiv.org/abs/2404.03411 · HTML: https://mlsanity.com/paper/2404.03411/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- First posted on arXiv 4 Apr 2024.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Built on by (most cited first, 30 of 34)

- [Improved Techniques for Optimization-Based Jailbreaking on Large Language Models](https://mlsanity.com/paper/2405.21018/) (2024)
- [A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends](https://mlsanity.com/paper/2407.07403/) (2024)
- [Best-of-N Jailbreaking](https://mlsanity.com/paper/2412.03556/) (2024)
- [A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations](https://mlsanity.com/paper/2502.14881/) (2025)
- [Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attacks](https://mlsanity.com/paper/2502.13175/) (2025)
- [MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialogue](https://mlsanity.com/paper/2411.03814/) (2024)
- [JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models](https://mlsanity.com/paper/2406.09321/) (2024)
- [Failures to Find Transferable Image Jailbreaks Between Vision-Language Models](https://mlsanity.com/paper/2407.15211/) (2024)
- [ShieldGemma 2: Robust and Tractable Image Content Moderation](https://mlsanity.com/paper/2504.01081/) (2025)
- [Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Model](https://mlsanity.com/paper/2406.15279/) (2024)
- [From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking](https://mlsanity.com/paper/2406.14859/) (2024)
- [Adversarial Attacks of Vision Tasks in the Past 10 Years: A Survey](https://mlsanity.com/paper/2410.23687/) (2024)
- [Large Multi-modal Models Can Interpret Features in Large Multi-modal Models](https://mlsanity.com/paper/2411.14982/) (2024)
- [Aloe: A Family of Fine-tuned Open Healthcare LLMs](https://mlsanity.com/paper/2405.01886/) (2024)
- [HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessment](https://mlsanity.com/paper/2509.24384/) (2025)
- [Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs](https://mlsanity.com/paper/2408.01355/) (2024)
- [Multimodal Pragmatic Jailbreak on Text-to-image Models](https://mlsanity.com/paper/2409.19149/) (2024)
- [LLM Jailbreak Detection for (Almost) Free!](https://mlsanity.com/paper/2509.14558/) (2025)
- [Transferable Adversarial Attacks on Black-Box Vision-Language Models](https://mlsanity.com/paper/2505.01050/) (2025)
- [A Survey on Responsible Generative AI: What to Generate and What Not](https://mlsanity.com/paper/2404.05783/) (2024)
- [Boosting Jailbreak Transferability for Large Language Models](https://mlsanity.com/paper/2410.15645/) (2024)
- [REVEAL: Multi-turn Evaluation of Image-Input Harms for Vision LLM](https://mlsanity.com/paper/2505.04673/) (2025)
- [Can an Individual Manipulate the Collective Decisions of Multi-Agents?](https://mlsanity.com/paper/2509.16494/) (2025)
- [Global Challenge for Safe and Secure LLMs Track 1](https://mlsanity.com/paper/2411.14502/) (2024)
- [Efficient Safety Retrofitting Against Jailbreaking for LLMs](https://mlsanity.com/paper/2502.13603/) (2025)
- [MetaCipher: A Time-Persistent and Universal Multi-Agent Framework for Cipher-Based Jailbreak Attacks for LLMs](https://mlsanity.com/paper/2506.22557/) (2025)
- [Self-Aware Safety Augmentation: Leveraging Internal Semantic Understanding to Enhance Safety in Vision-Language Models](https://mlsanity.com/paper/2507.21637/) (2025)
- [Self-adaptive Dataset Construction for Real-World Multimodal Safety Scenarios](https://mlsanity.com/paper/2509.04403/) (2025)
- [Deep Research Agents Brings Deeper Harm](https://mlsanity.com/paper/2510.11851/) (2025)
- [CFSafety: Comprehensive Fine-grained Safety Assessment for LLMs](https://mlsanity.com/paper/2410.21695/) (2024)

## Abstract (arXiv)

Various jailbreak attacks have been proposed to red-team Large Language Models (LLMs) and revealed the vulnerable safeguards of LLMs. Besides, some methods are not limited to the textual modality and extend the jailbreak attack to Multimodal Large Language Models (MLLMs) by perturbing the visual input. However, the absence of a universal evaluation benchmark complicates the performance reproduction and fair comparison. Besides, there is a lack of comprehensive evaluation of closed-source state-of-the-art (SOTA) models, especially MLLMs, such as GPT-4V. To address these issues, this work first builds a comprehensive jailbreak evaluation dataset with 1445 harmful questions covering 11 different safety policies. Based on this dataset, extensive red-teaming experiments are conducted on 11 different LLMs and MLLMs, including both SOTA proprietary models and open-source models. We then conduct a deep analysis of the evaluated results and find that (1) GPT4 and GPT-4V demonstrate better robustness against jailbreak attacks compared to open-source LLMs and MLLMs. (2) Llama2 and Qwen-VL-Chat are more robust compared to other open-source models. (3) The transferability of visual jailbreak methods is relatively limited compared to textual jailbreak methods. The dataset and code can be found https://github.com/chenxshuo/RedTeamingGPT4V