mlsanityData to 14 Oct 2026

VLMGuard-R1: Proactive Safety Alignment for VLMs via Reasoning-Driven Prompt Optimization

arXiv 2504.12661 ↗First posted 17 Apr 2025cs.LG cs.CL cs.CVcited by 10

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 3 more
Abstract (arXiv)

Aligning Vision-Language Models (VLMs) with safety standards is essential to mitigate risks arising from their multimodal complexity, where integrating vision and language unveils subtle threats beyond the reach of conventional safeguards. Inspired by the insight that reasoning across modalities is key to preempting intricate vulnerabilities, we propose a novel direction for VLM safety: multimodal reasoning-driven prompt rewriting. To this end, we introduce VLMGuard-R1, a proactive framework that refines user inputs through a reasoning-guided rewriter, dynamically interpreting text-image interactions to deliver refined prompts that bolster safety across diverse VLM architectures without altering their core parameters. To achieve this, we devise a three-stage reasoning pipeline to synthesize a dataset that trains the rewriter to infer subtle threats, enabling tailored, actionable responses over generic refusals. Extensive experiments across three benchmarks with five VLMs reveal that VLMGuard-R1 outperforms four baselines. In particular, VLMGuard-R1 achieves a remarkable 43.59% increase in average safety across five models on the SIUO benchmark.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 26 of its 38 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

26 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models30 Jan
  2. 2024OpenAI o1 System Card21 Dec

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  9. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  10. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  11. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 454
  12. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
12 more
  1. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  2. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  3. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 117
  4. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 116
  5. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 85
  6. 2024Vision-Language Models Can Self-Improve Reasoning via Reflectioncited by 69
  7. 2024Red Teaming Visual Language Modelscited by 65
  8. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safetycited by 63
  9. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Timecited by 60
  10. 2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibrationcited by 31
  11. 2024Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Modelcited by 26
  12. 2024From Text to Transformation: A Comprehensive Review of Large Language Models' Versatilitycited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Modelscited by 10
  2. 2025DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Promptcited by 8
  3. 2025EcoAlign: An Economically Rational Framework for Efficient LVLM Alignmentcited by 5
  4. 2025A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challengescited by 3
  5. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  6. 2026GuardAlign: Test-time Safety Alignment in Multimodal Large Language Modelscited by 1
  7. 2026Steering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routingcited by 0
  8. 2026PolicyShiftGuard: Benchmarking and Improving Policy-Adaptive Image Guardrailscited by 0
  9. 2026Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Modelscited by 0
  10. 2026Can Vision-Language Models Stay Helpful When Facing Implicit Risks? Intent-Privilege OPSD for Efficient Safety-Helpfulness Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap
  2. 2025PRISM: Robust VLM Alignment with Principled Reasoning for Integrated Safety in Multimodality
  3. 2025MSR-Align: Policy-Grounded Multimodal Alignment for Safety-Aware Reasoning in Vision-Language Models
  4. 2025Think-Reflect-Revise: A Policy-Guided Reflective Framework for Safety Alignment in Large Vision Language Models
  5. 2025VisuoAlign: Safety Alignment of LVLMs with Multimodal Tree Search
  6. 2025SHIELD: Classifier-Guided Prompting for Robust and Safer LVLMs
  7. 2025The Safety Reminder: A Soft Prompt to Reactivate Delayed Safety Awareness in Vision-Language Models
  8. 2025SafeCoT: Improving VLM Safety with Minimal Reasoning