# VLMGuard-R1: Proactive Safety Alignment for VLMs via Reasoning-Driven Prompt Optimization

arXiv: https://arxiv.org/abs/2504.12661 · HTML: https://mlsanity.com/paper/2504.12661/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 17 Apr 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)
- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [LLaVA-CoT: Let Vision Language Models Reason Step-by-Step](https://mlsanity.com/paper/2411.10440/) (2024)
- [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023)
- [InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output](https://mlsanity.com/paper/2407.03320/) (2024)
- [Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models](https://mlsanity.com/paper/2402.02207/) (2024)
- [MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance](https://mlsanity.com/paper/2401.02906/) (2024)
- [Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2411.14432/) (2024)
- [SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model](https://mlsanity.com/paper/2406.12030/) (2024)
- [Vision-Language Models Can Self-Improve Reasoning via Reflection](https://mlsanity.com/paper/2411.00855/) (2024)
- [Red Teaming Visual Language Models](https://mlsanity.com/paper/2401.12915/) (2024)
- [VLSBench: Unveiling Visual Leakage in Multimodal Safety](https://mlsanity.com/paper/2411.19939/) (2024)
- [ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time](https://mlsanity.com/paper/2410.06625/) (2024)
- [CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration](https://mlsanity.com/paper/2409.11365/) (2024)
- [Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Model](https://mlsanity.com/paper/2406.15279/) (2024)
- [Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models](https://mlsanity.com/paper/2501.18533/) (2025)
- [From Text to Transformation: A Comprehensive Review of Large Language Models' Versatility](https://mlsanity.com/paper/2402.16142/) (2024)

## Built on by (most cited first, 10 of 10)

- [SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Models](https://mlsanity.com/paper/2510.06871/) (2025)
- [DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Prompt](https://mlsanity.com/paper/2506.09353/) (2025)
- [EcoAlign: An Economically Rational Framework for Efficient LVLM Alignment](https://mlsanity.com/paper/2511.11301/) (2025)
- [A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challenges](https://mlsanity.com/paper/2507.21117/) (2025)
- [Empowering Multimodal LLMs with External Tools: A Comprehensive Survey](https://mlsanity.com/paper/2508.10955/) (2025)
- [GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models](https://mlsanity.com/paper/2602.24027/) (2026)
- [Steering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routing](https://mlsanity.com/paper/2603.15647/) (2026)
- [PolicyShiftGuard: Benchmarking and Improving Policy-Adaptive Image Guardrails](https://mlsanity.com/paper/2607.05910/) (2026)
- [Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models](https://mlsanity.com/paper/2608.27531/) (2026)
- [Can Vision-Language Models Stay Helpful When Facing Implicit Risks? Intent-Privilege OPSD for Efficient Safety-Helpfulness Alignment](https://mlsanity.com/paper/2609.37837/) (2026)

## Abstract (arXiv)

Aligning Vision-Language Models (VLMs) with safety standards is essential to mitigate risks arising from their multimodal complexity, where integrating vision and language unveils subtle threats beyond the reach of conventional safeguards. Inspired by the insight that reasoning across modalities is key to preempting intricate vulnerabilities, we propose a novel direction for VLM safety: multimodal reasoning-driven prompt rewriting. To this end, we introduce VLMGuard-R1, a proactive framework that refines user inputs through a reasoning-guided rewriter, dynamically interpreting text-image interactions to deliver refined prompts that bolster safety across diverse VLM architectures without altering their core parameters. To achieve this, we devise a three-stage reasoning pipeline to synthesize a dataset that trains the rewriter to infer subtle threats, enabling tailored, actionable responses over generic refusals. Extensive experiments across three benchmarks with five VLMs reveal that VLMGuard-R1 outperforms four baselines. In particular, VLMGuard-R1 achieves a remarkable 43.59% increase in average safety across five models on the SIUO benchmark.