mlsanityData to 7 Oct 2026

Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projection

arXiv 2603.14825 ↗First posted 16 Mar 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Existing jailbreak defence frameworks for Large Vision-Language Models often suffer from a safety utility tradeoff, where strengthening safety inadvertently degrades performance on general visual-grounded reasoning tasks. In this work, we investigate whether safety and utility are inherently antagonistic objectives. We focus on a modality induced bias direction consistently observed across datasets, which arises from suboptimal coupling between the Large Language Model backbone and visual encoders. We further demonstrate that this direction undermines performance on both tasks. Leveraging this insight, we propose Two Birds, One Projection, an efficient inference time jailbreak defence that projects cross-modal features onto the null space of the identified bias direction to remove the corresponding components. Requiring only a single forward pass, our method effectively breaks the conventional tradeoff, simultaneously improving both safety and utility across diverse benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 52 references that are in the map. First posted on arXiv 16 Mar 2026.

Builds on

41 of 52 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models25 Apr
  2. 2025Understanding and Rectifying Safety Perception Distortion in VLMs18 Feb
  3. 2025Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency9 Jan
  4. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment27 Nov
  5. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks23 Nov
  6. 2024BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks28 Oct
  7. 2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models11 Oct
  8. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time9 Oct
  9. 2024Refusing Safe Prompts for Multi-modal Large Language Models12 Jul
  10. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt6 Jun
  11. 2024White-box Multimodal Jailbreaks Against Large Vision-Language Models28 May
  12. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models14 Mar
8 more
  1. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
  2. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting14 Mar
  3. 2024Test-Time Backdoor Attacks on Multimodal Large Language Models13 Feb
  4. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance5 Jan
  5. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov
  6. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  7. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  8. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  9. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  11. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  12. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
9 more
  1. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  2. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  3. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 78
  4. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  5. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 62
  6. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
  7. 2025Steer LLM Latents for Hallucination Detectioncited by 49
  8. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  9. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Safe Semantics, Unsafe Interpretations: Tackling Implicit Reasoning Safety in Large Vision-Language Models
  2. 2025Security Tensors as a Cross-Modal Bridge: Extending Text-Aligned Safety to Vision in LVLM
  3. 2025JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models
  4. 2025Think-Reflect-Revise: A Policy-Guided Reflective Framework for Safety Alignment in Large Vision Language Models
  5. 2026Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection
  6. 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models
  7. 2025VisuoAlign: Safety Alignment of LVLMs with Multimodal Tree Search
  8. 2026Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility