# JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steering

arXiv: https://arxiv.org/abs/2508.05087 · HTML: https://mlsanity.com/paper/2508.05087/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 17 of 17)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023)
- [Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models](https://mlsanity.com/paper/2307.14539/) (2023)
- [JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2404.03027/) (2024)
- [AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting](https://mlsanity.com/paper/2403.09513/) (2024)
- [Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation](https://mlsanity.com/paper/2403.09572/) (2024)
- [Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking](https://mlsanity.com/paper/2311.09827/) (2023)
- [Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Character](https://mlsanity.com/paper/2405.20773/) (2024)
- [VLSBench: Unveiling Visual Leakage in Multimodal Safety](https://mlsanity.com/paper/2411.19939/) (2024)
- [Jailbreak Large Vision-Language Models Through Multi-Modal Linkage](https://mlsanity.com/paper/2412.00473/) (2024)
- [Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Prompts](https://mlsanity.com/paper/2407.15050/) (2024)
- [Give Me Your Attention: Dot-Product Attention Considered Harmful for Adversarial Patch Robustness](https://mlsanity.com/paper/2203.13639/) (2022)
- [Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities](https://mlsanity.com/paper/2410.18469/) (2024)
- [TIDE : Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generation](https://mlsanity.com/paper/2503.07050/) (2025)
- [Image-to-Text Logic Jailbreak: Your Imagination can Help You Do Anything](https://mlsanity.com/paper/2407.02534/) (2024)

## Built on by (most cited first, 11 of 11)

- [OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluation](https://mlsanity.com/paper/2512.06589/) (2025)
- [Making MLLMs Blind: Adversarial Smuggling Attacks in MLLM Content Moderation](https://mlsanity.com/paper/2604.06950/) (2026)
- [Furina: Fragmented Uncertainty-Driven Refusal Instability Attack](https://mlsanity.com/paper/2605.26158/) (2026)
- [VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Models](https://mlsanity.com/paper/2510.17759/) (2025)
- [SafeThinker: Reasoning about Risk to Deepen Safety Beyond Shallow Alignment](https://mlsanity.com/paper/2601.16506/) (2026)
- [Hard to Read, Easy to Jailbreak: How Visual Degradation Bypasses MLLM Safety Alignment](https://mlsanity.com/paper/2605.07250/) (2026)
- [Mosaic: Multimodal Jailbreak against Closed-Source VLMs via Multi-View Ensemble Optimization](https://mlsanity.com/paper/2604.09253/) (2026)
- [Localize and Neutralize: Gradient-guided Token Suppression against Visual Prompt Injection Attack](https://mlsanity.com/paper/2605.25194/) (2026)
- [Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation](https://mlsanity.com/paper/2608.03166/) (2026)
- [Repeat-After-Me: Black-Box Adaptive Visual Prompt Injection](https://mlsanity.com/paper/2609.04533/) (2026)
- [Scenes as an Attack Surface: Uncovering Scene-Grounded Safety Failures in MLLMs](https://mlsanity.com/paper/2610.14094/) (2026)

## Abstract (arXiv)

Jailbreak attacks against multimodal large language Models (MLLMs) are a significant research focus. Current research predominantly focuses on maximizing attack success rate (ASR), often overlooking whether the generated responses actually fulfill the attacker's malicious intent. This oversight frequently leads to low-quality outputs that bypass safety filters but lack substantial harmful content. To address this gap, we propose JPS, underlineJailbreak MLLMs with collaborative visual underlinePerturbation and textual underlineSteering, which achieves jailbreaks via corporation of visual image and textually steering prompt. Specifically, JPS utilizes target-guided adversarial image perturbations for effective safety bypass, complemented by "steering prompt" optimized via a multi-agent system to specifically guide LLM responses fulfilling the attackers' intent. These visual and textual components undergo iterative co-optimization for enhanced performance. To evaluate the quality of attack outcomes, we propose the Malicious Intent Fulfillment Rate (MIFR) metric, assessed using a Reasoning-LLM-based evaluator. Our experiments show JPS sets a new state-of-the-art in both ASR and MIFR across various MLLMs and benchmarks, with analyses confirming its efficacy. Codes are available at hrefhttps://github.com/thu-coai/JPShttps://github.com/thu-coai/JPS. colorwarningcolorWarning: This paper contains potentially sensitive contents.