mlsanityData to 14 Oct 2026

Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback

arXiv 2503.17682 ↗First posted 22 Mar 2025cs.LG cs.AIcited by 30

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) are essential for building general-purpose AI assistants; however, they pose increasing safety risks. How can we ensure safety alignment of MLLMs to prevent undesired behaviors? Going further, it is critical to explore how to fine-tune MLLMs to preserve capabilities while meeting safety constraints. Fundamentally, this challenge can be formulated as a min-max optimization problem. However, existing datasets have not yet disentangled single preference signals into explicit safety constraints, hindering systematic investigation in this direction. Moreover, it remains an open question whether such constraints can be effectively incorporated into the optimization process for multi-modal models. In this work, we present the first exploration of the Safe RLHF-V -- the first multimodal safety alignment framework. The framework consists of: (I) BeaverTails-V, the first open-source dataset featuring dual preference annotations for helpfulness and safety, supplemented with multi-level safety labels (minor, moderate, severe); (II) Beaver-Guard-V, a multi-level guardrail system to proactively defend against unsafe queries and adversarial attacks. Applying the guard model over five rounds of filtering and regeneration significantly enhances the precursor model's overall safety by an average of 40.9%. (III) Based on dual preference, we initiate the first exploration of multi-modal safety alignment within a constrained optimization. Experimental results demonstrate that Safe RLHF effectively improves both model helpfulness and safety. Specifically, Safe RLHF-V enhances model safety by 34.2% and helpfulness by 34.3%.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 73 references that are in the map. First posted on arXiv 22 Mar 2025.

Builds on

46 of 73 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations15 Nov
  2. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model17 Jun
  3. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models11 Jun
  4. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
  5. 2024SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection5 Mar
  6. 2024Safety of Multimodal Large Language Models on Images and Texts1 Feb
  7. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance5 Jan
  8. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov

Other references · 38

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
26 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  2. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  5. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  8. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  9. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  10. 2023AI Alignment: A Comprehensive Surveycited by 263
  11. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  12. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  13. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  14. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  15. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  16. 2024Aligner: Efficient Alignment by Learning to Correctcited by 98
  17. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 92
  18. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  19. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 83
  20. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  21. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 72
  22. 2024UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Imagescited by 53
  23. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 39
  24. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!cited by 37
  25. 2024SafeInfer: Context Adaptive Decoding Time Safety Alignment for Large Language Modelscited by 23
  26. 2024MLLM-as-a-Judge for Image Safety without Human Labelingcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 20.

30 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  2. 2025Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approachcited by 24
  3. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 11
  4. 2025SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Modelscited by 10
  5. 2026SingGuard: A Policy-Adaptive Multimodal LLM Guardrail with Dynamic Reasoningcited by 6
  6. 2026Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memorycited by 5
  7. 2025ProGuard: Towards Proactive Multimodal Safeguardcited by 4
  8. 2026SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cuescited by 4
  9. 2026Pragma-VL: Towards a Pragmatic Arbitration of Safety and Helpfulness in MLLMscited by 3
  10. 2025Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrixcited by 2
  11. 2025Beyond Text: Multimodal Jailbreaking of Vision-Language and Audio Models through Perceptually Simple Transformationscited by 1
  12. 2025Debate with Images: Detecting Deceptive Behaviors in Multimodal Large Language Modelscited by 1
18 more
  1. 2025Think-Reflect-Revise: A Policy-Guided Reflective Framework for Safety Alignment in Large Vision Language Modelscited by 1
  2. 2026SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamicscited by 1
  3. 2026Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correctioncited by 1
  4. 2026Constitutional On-Policy Safe Distillationcited by 1
  5. 2025When Slower Isn't Truer: Inverse Scaling Law of Truthfulness in Multimodal Reasoningcited by 0
  6. 2026Towards Multi-modal Multi-turn Safety: From Agentic Interaction to Strategic Alignmentcited by 0
  7. 2026From Sparse Decisions to Dense Reasoning: A Multi-attribute Trajectory Paradigm for Multimodal Moderationcited by 0
  8. 2026SaFeR-ToolKit: Structured Reasoning via Virtual Tool Calling for Multimodal Safetycited by 0
  9. 2026OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequencescited by 0
  10. 2026You Snooze, You Lose: Automatic Safety Alignment Restoration through Neural Weight Translationcited by 0
  11. 2026Safe Alone, Unsafe Together: Safeguarding Against Implicit Toxicity When Benign Images Combinecited by 0
  12. 2026Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Modelscited by 0
  13. 2026SafeAtlas-VL: Beyond Binary Multimodal Safety with Large-Scale Data and Guard Modelscited by 0
  14. 2026Do VLMs Share Safety Neurons Across Modalities?cited by 0
  15. 2026VisionPsy-Nano: Improving Accuracy, Efficiency, and Reliability in On-Device Vision-Language Modelscited by 0
  16. 2026Can Vision-Language Models Stay Helpful When Facing Implicit Risks? Intent-Privilege OPSD for Efficient Safety-Helpfulness Alignmentcited by 0
  17. 2026Visual Grounding Safety in Vision-Language Modelscited by 0
  18. 2026VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning from Human Feedback with High-Confidence Safety Constraints
  2. 2025Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models
  3. 2024Cross-Modal Safety Alignment: Is textual unlearning all you need?
  4. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards
  5. 2024Evaluating Defences against Unsafe Feedback in RLHF
  6. 2025Breaking the Safety-Capability Tradeoff: Reinforcement Learning with Verifiable Rewards Maintains Safety Guardrails in LLMs
  7. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety
  8. 2025When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal Large Language Models