mlsanityData to 14 Oct 2026

When Seeing Overrides Knowing: Visual Dominance and Deferral-Based Method for Personalized Safety in VLMs

arXiv 2609.04281 ↗First posted 3 Sep 2026cs.CV cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language models (VLMs) are increasingly deployed in high-stakes settings, where a response that is reasonable in general may still be unsafe for a particular user whose medical, emotional, or situational context is unknown to the model. We study this problem of personalized safety in multimodal systems and introduce MPS-Bench, a benchmark of 5,181 scenarios from 584 real-world images across 12 high-risk domains, each paired with a hidden user profile. Evaluating eight frontier VLMs, we find that they almost always respond directly (86-99%) rather than seek missing context, and none exceeds 2.6/5 on personalized safety. To understand why these failures arise, we analyze multimodal interactions and identify visual dominance: visual information enters text representations early and suppresses textual risk signals during multimodal fusion. Causal interventions reveal a two-stage mechanism in which visual affect is first transferred into the text stream in early layers and then shapes the final decision through this altered text representation, making late-stage internal remediation unreliable. Motivated by this mechanism, we propose PRISM, a lightweight input monitor that uses bidirectional cross-modal modulation to predict when a query is likely to require deferral. PRISM achieves 0.978 AUC and strictly dominates the safety-utility Pareto frontier across all tested models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 50 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

30 of 50 references.

References · 30

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  3. 2022Red Teaming Language Models with Language Modelscited by 1,026
  4. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  5. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  6. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  7. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 433
  8. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  9. 2023Large Language Models in Finance: A Surveycited by 226
  10. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  11. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 146
  12. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 125
18 more
  1. 2025AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agentscited by 90
  2. 2018CrisisMMD: Multimodal Twitter Datasets from Natural Disasterscited by 89
  3. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 89
  4. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 87
  5. 2016Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmarkcited by 71
  6. 2023Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Datacited by 66
  7. 2024The opportunities and risks of large language models in mental healthcited by 54
  8. 2024Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?cited by 24
  9. 2017Protest Activity Detection and Perceived Violence Estimation from Social Media Imagescited by 20
  10. 2024ChatDiet: Empowering Personalized Nutrition-Oriented Food Recommender Chatbots through an LLM-Augmented Frameworkcited by 20
  11. 2025When Language Overrules: Revealing Text Dominance in Multimodal Large Language Modelscited by 18
  12. 2025Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performancecited by 18
  13. 2025Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent Approachcited by 16
  14. 2025Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?cited by 14
  15. 2025Is Safety Standard Same for Everyone? User-Specific Safety Evaluation of Large Language Modelscited by 10
  16. 2023Integrated Image and Location Analysis for Wound Classification: A Deep Learning Approachcited by 9
  17. 2024Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision-Language Modelscited by 7
  18. 2021A Dataset and System for Real-Time Gun Detection in Surveillance Video Using Deep Learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images
  2. 2026When Helpers Become Hazards: A Benchmark for Analyzing Multimodal LLM-Powered Safety in Daily Life
  3. 2025MSTS: A Multimodal Safety Test Suite for Vision-Language Models
  4. 2026Counterfactual Neighborhood Risk Fields: Evidence-Grounded Selective Safety for Vision-Language Models
  5. 2024Multimodal Situational Safety
  6. 2025VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety
  7. 2025RedDiffuser: Auditing Multimodal Safety Failures in Vision-Language Models via Reinforced Diffusion
  8. 2026SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues