mlsanityData to 14 Oct 2026

Learning To See But Forgetting To Follow: Visual Instruction Tuning Makes LLMs More Prone To Jailbreak Attacks

arXiv 2405.04403 ↗First posted 7 May 2024cs.CV cs.CLcited by 15

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Augmenting Large Language Models (LLMs) with image-understanding capabilities has resulted in a boom of high-performing Vision-Language models (VLMs). While studying the alignment of LLMs to human values has received widespread attention, the safety of VLMs has not received the same attention. In this paper, we explore the impact of jailbreaking on three state-of-the-art VLMs, each using a distinct modeling approach. By comparing each VLM to their respective LLM backbone, we find that each VLM is more susceptible to jailbreaking. We consider this as an undesirable outcome from visual instruction-tuning, which imposes a forgetting effect on an LLM's safety guardrails. Therefore, we provide recommendations for future work based on evaluation strategies that aim to highlight the weaknesses of a VLM, as well as take safety measures into account during visual instruction tuning.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 68 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

38 of 68 references.

References · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  8. 2013Intriguing properties of neural networkscited by 5,155
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2017Deep reinforcement learning from human preferencescited by 3,912
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
26 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  6. 2019Towards VQA Models That Can Readcited by 1,718
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  11. 2022Red Teaming Language Models with Language Modelscited by 1,026
  12. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  13. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  14. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  15. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  16. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  17. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  18. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  19. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  20. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  21. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  22. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  23. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 115
  24. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 99
  25. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  26. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 4.

15 papers.

Sort:

By area:

  1. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 67
  2. 2024Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Modelcited by 26
  3. 2025Lessons From Red Teaming 100 Generative AI Productscited by 23
  4. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustnesscited by 10
  5. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 10
  6. 2024How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?cited by 8
  7. 2024Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modelingcited by 6
  8. 2026Jailbreaking LLMs & VLMs: Mechanisms, Evaluation, and Unified Defensecited by 3
  9. 2026When Helpers Become Hazards: A Benchmark for Analyzing Multimodal LLM-Powered Safety in Daily Lifecited by 3
  10. 2025Self-adaptive Dataset Construction for Real-World Multimodal Safety Scenarioscited by 2
  11. 2025Sequential Comics for Jailbreaking Multimodal Large Language Models via Structured Visual Storytellingcited by 2
  12. 2025VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Modelscited by 1
3 more
  1. 2026PC2: Politically Controversial Content Generation via Jailbreaking Attacks on GPT-based Text-to-Image Modelscited by 1
  2. 2026SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learningcited by 1
  3. 2025Text2VLM: Adapting Text-Only Datasets to Evaluate Alignment Training in Visual Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models
  2. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense
  3. 2026Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift
  4. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image
  5. 2026MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models
  6. 2026Jailbreaks on Vision Language Model via Multimodal Reasoning
  7. 2026Jailbreaking Vision-Language Models Through the Visual Modality
  8. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden States