mlsanityData to 7 Oct 2026

Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models

arXiv 2402.02207 ↗First posted 3 Feb 2024cs.LGcited by 160

On the map: LLM safety and security → Vision-language and text-to-image safety

Abstract (arXiv)

Current vision large language models (VLLMs) exhibit remarkable capabilities yet are prone to generate harmful content and are vulnerable to even the simplest jailbreaking attacks. Our initial analysis finds that this is due to the presence of harmful data during vision-language instruction fine-tuning, and that VLLM fine-tuning can cause forgetting of safety alignment previously learned by the underpinning LLM. To address this issue, we first curate a vision-language safe instruction-following dataset VLGuard covering various harmful categories. Our experiments demonstrate that integrating this dataset into standard vision-language fine-tuning or utilizing it for post-hoc fine-tuning effectively safety aligns VLLMs. This alignment is achieved with minimal impact on, or even enhancement of, the models' helpfulness. The versatility of our safety fine-tuning dataset makes it a valuable resource for safety-testing existing VLLMs, training new models or safeguarding pre-trained VLLMs. Empirical results demonstrate that fine-tuned VLLMs effectively reject unsafe instructions and substantially reduce the success rates of several black-box adversarial attacks, which approach zero in many cases. The code and dataset are available at https://github.com/ys-zong/VLGuard.

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security. Builds on 33 of its 47 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

33 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
20 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  2. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  4. 2023A Survey on Multimodal Large Language Modelscited by 898
  5. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  6. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  8. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  9. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  11. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  12. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  13. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  14. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  15. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  16. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  17. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  18. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  19. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  20. 2023HOD: A Benchmark Dataset for Harmful Object Detectioncited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 43; 2025: 67; 2026: 50.

160 papers.

Sort:

By area:

  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  2. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  3. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
  4. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 125
  5. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 114
  6. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 113
  7. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  8. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  9. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  10. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
  11. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  12. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 82
38 more
  1. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  2. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  3. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  4. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  5. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 61
  6. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Timecited by 57
  7. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreakscited by 44
  8. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  9. 2025Jailbreaking Multimodal Large Language Models via Shuffle Inconsistencycited by 42
  10. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
  11. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  12. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  13. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Drivingcited by 35
  14. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 34
  15. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  16. 2024Self-Supervised Visual Preference Alignmentcited by 31
  17. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  18. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  19. 2024Safety Alignment for Vision Language Modelscited by 27
  20. 2025Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedbackcited by 27
  21. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  22. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25
  23. 2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Modelscited by 23
  24. 2024Cross-Modal Safety Alignment: Is textual unlearning all you need?cited by 22
  25. 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Modelscited by 21
  26. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defensecited by 20
  27. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  28. 2025Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligencecited by 20
  29. 2024RapGuard: Safeguarding Multimodal Large Language Models via Rationale-aware Defensive Promptingcited by 19
  30. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  31. 2024Cross-Modal Safety Mechanism Transfer in Large Vision-Language Modelscited by 18
  32. 2024UniGuard: Towards Universal Safety Guardrails for Jailbreak Attacks on Multimodal Large Language Modelscited by 18
  33. 2025Think in Safety: Unveiling and Mitigating Safety Alignment Collapse in Multimodal Large Reasoning Modelcited by 18
  34. 2024VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wildcited by 17
  35. 2025Understanding and Rectifying Safety Perception Distortion in VLMscited by 17
  36. 2025Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearningcited by 16
  37. 2024Jigsaw Puzzles: Splitting Harmful Questions to Jailbreak Large Language Modelscited by 15
  38. 2024FMM-Attack: A Flow-based Multi-modal Adversarial Attack on Video-based LLMscited by 14

50 most cited of 160 All 160 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Safe Vision-Language Models via Unsafe Weights Manipulation
  2. 2025VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap
  3. 2024Learning To See But Forgetting To Follow: Visual Instruction Tuning Makes LLMs More Prone To Jailbreak Attacks
  4. 2024PSA-VLM: Enhancing Vision-Language Model Safety through Progressive Concept-Bottleneck-Driven Alignment
  5. 2026Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking
  6. 2025Secure LLM Fine-Tuning via Safety-Aware Probing
  7. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
  8. 2026Understanding and Preserving Safety in Fine-Tuned LLMs