mlsanityData to 7 Oct 2026

Steering Multimodal Large Language Models Decoding for Context-Aware Safety

arXiv 2509.19212 ↗First posted 23 Sep 2025cs.CL cs.AIcited by 6

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) are increasingly deployed in real-world applications, yet their ability to make context-aware safety decisions remains limited. Existing methods often fail to balance oversensitivity (unjustified refusals of benign queries) and undersensitivity (missed detection of visually grounded risks), leaving a persistent gap in safety alignment. To address this issue, we introduce Safety-aware Contrastive Decoding (SafeCoDe), a lightweight and model-agnostic decoding framework that dynamically adjusts token generation based on multimodal context. SafeCoDe operates in two stages: (1) a contrastive decoding mechanism that highlights tokens sensitive to visual context by contrasting real and Gaussian-noised images, and (2) a global-aware token modulation strategy that integrates scene-level reasoning with token-level adjustment to adapt refusals according to the predicted safety verdict. Extensive experiments across diverse MLLM architectures and safety benchmarks, covering undersensitivity, oversensitivity, and general safety evaluations, show that SafeCoDe consistently improves context-sensitive refusal behaviors while preserving model helpfulness.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 81 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

66 of 81 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment27 Nov
  2. 2024Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench29 Oct
  3. 2024Multimodal Situational Safety8 Oct
  4. 2024MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries?22 Jun
  5. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models14 Mar
  6. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting14 Mar
  7. 2024Safety of Multimodal Large Language Models on Images and Texts1 Feb
  8. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov
  9. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  10. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  11. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 55

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2015VQA: Visual Question Answeringcited by 2,489
38 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  5. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  7. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  10. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  12. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  13. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  14. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  15. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  16. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  17. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  18. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  19. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  20. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  21. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  22. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  23. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  24. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 197
  25. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  26. 2023UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANscited by 165
  27. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  28. 2024Towards Safer Large Language Models through Machine Unlearningcited by 134
  29. 2016Visual Question Answering: A Survey of Methods and Datasetscited by 123
  30. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  31. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  32. 2024Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuningcited by 87
  33. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  34. 2023Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Modelscited by 56
  35. 2024VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videoscited by 51
  36. 2025From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Trainingcited by 49
  37. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  38. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 29

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Steered LLM Activations are Non-Surjectivecited by 7
  2. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialoguescited by 4
  3. 2026AMPS: Adaptive Modality Preference Steering via Functional Entropycited by 4
  4. 2026Prompt-Activation Duality: Improving Activation Steering via Attention-Level Interventionscited by 3
  5. 2026AOR-Bench: Do Large Audio Language Models Over-Refuse Pseudo-Harmful Queries?cited by 0
  6. 2026Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs
  2. 2026Transfer Safety Awareness for Cross-Modal Safety Drift in Multimodal Large Language Models
  3. 2026SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models
  4. 2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration
  5. 2026COMIC: Reference-Aware Safety Gating for Multimodal Large Language Models
  6. 2026Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory
  7. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation
  8. 2026CASA: Classification Augmented with Safety Attention for Robust Multimodal Alignment