mlsanityData to 14 Oct 2026

FORCE: Transferable Visual Jailbreaking Attacks via Feature Over-Reliance CorrEction

arXiv 2509.21029 ↗First posted 25 Sep 2025cs.LGcited by 11

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

The integration of new modalities enhances the capabilities of multimodal large language models (MLLMs) but also introduces additional vulnerabilities. In particular, simple visual jailbreaking attacks can manipulate open-source MLLMs more readily than sophisticated textual attacks. However, these underdeveloped attacks exhibit extremely limited cross-model transferability, failing to reliably identify vulnerabilities in closed-source MLLMs. In this work, we analyse the loss landscape of these jailbreaking attacks and find that the generated attacks tend to reside in high-sharpness regions, whose effectiveness is highly sensitive to even minor parameter changes during transfer. To further explain the high-sharpness localisations, we analyse their feature representations in both the intermediate layers and the spectral domain, revealing an improper reliance on narrow layer representations and semantically poor frequency components. Building on this, we propose a Feature Over-Reliance CorrEction (FORCE) method, which guides the attack to explore broader feasible regions across layer features and rescales the influence of frequency features according to their semantic content. By eliminating non-generalizable reliance on both layer and spectral features, our method discovers flattened feasible regions for visual jailbreaking attacks, thereby improving cross-model transferability. Extensive experiments demonstrate that our approach effectively facilitates visual red-teaming evaluations against closed-source MLLMs.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 63 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

50 of 63 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Distraction is All You Need for Multimodal Large Language Model Jailbreaking15 Feb
  2. 2025Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency9 Jan
  3. 2024Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models27 Nov
  4. 2024Failures to Find Transferable Image Jailbreaks Between Vision-Language Models21 Jul
  5. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models14 Mar
  6. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  7. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
  8. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  10. 2023Mistral 7Bcited by 3,164
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
30 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  4. 2022Red Teaming Language Models with Language Modelscited by 1,026
  5. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  8. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
  9. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  10. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  11. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  12. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  13. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  14. 2024OmniGen: Unified Image Generationcited by 394
  15. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  16. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  17. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 172
  18. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 172
  19. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 135
  20. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attackscited by 89
  21. 2024Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Modelscited by 33
  22. 2023Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learningcited by 27
  23. 2025Understanding and Enhancing the Transferability of Jailbreaking Attackscited by 22
  24. 2024LaVin-DiT: Large Vision Diffusion Transformercited by 21
  25. 2025Early Stopping Against Label Noise Without Validation Datacited by 21
  26. 2023Late Stopping: Avoiding Confidently Learning from Mislabeled Examplescited by 17
  27. 2023Sharpness-Aware Minimization Alone can Improve Adversarial Robustnesscited by 14
  28. 2025Guiding not Forcing: Enhancing the Transferability of Jailbreaking Attacks on LLMs via Removing Superfluous Constraintscited by 11
  29. 2022Exploring Adversarial Robustness of Vision Transformers in the Spectral Perspectivecited by 8
  30. 2024TED-VITON: Transformer-Empowered Diffusion Models for Virtual Try-Oncited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2026When Safety Collides: Resolving Multi-Category Harmful Conflicts in Text-to-Image Diffusion via Adaptive Safety Guidancecited by 11
  2. 2026VII: Visual Instruction Injection for Jailbreaking Image-to-Video Generation Modelscited by 7
  3. 2026Is Gradient Ascent Really Necessary? Memorize to Forget for Machine Unlearningcited by 6
  4. 2026Delving into the Temporal Challenges of Unified Video Protection Against Image-to-Video and Fine-Tuning-based Customizationcited by 5
  5. 2026Mobile-VTON: High-Fidelity On-Device Virtual Try-Oncited by 2
  6. 2026Visual Token Compression Enhances Robustness of MLLMscited by 2
  7. 2025High-Entropy Tokens as Multimodal Failure Points in Vision-Language Modelscited by 1
  8. 2026The Alignment Curse: Modality Alignment Supercharges Audio Attacks via Text Transfercited by 1
  9. 2026Mitigating Mismatch within Reference-based Preference Optimizationcited by 1
  10. 2026Break the Brake, Not the Wheel: Untargeted Jailbreak via Entropy Maximizationcited by 0
  11. 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models
  2. 2026Toward Universal and Transferable Jailbreak Attacks on Vision-Language Models
  3. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks
  4. 2025JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Model
  5. 2026Overloading Large Vision-Language Models for Jailbreaking
  6. 2025JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steering
  7. 2025Enhanced MLLM Black-Box Jailbreaking Attacks and Defenses
  8. 2025Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensity