mlsanityData to 7 Oct 2026

SPARK: Representation-Level KV Memory Alignment for Safer Vision-Language Models

arXiv 2609.14258 ↗First posted 13 Sep 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Vision-language models (VLMs) remain vulnerable to jailbreaks that distribute harmful intent across text and images, making unimodal safety mechanisms insufficient. We investigate whether this vulnerability can be mitigated directly in the multimodal key-value (KV) memory formed during prefill, without modifying model parameters at inference time. We introduce SPARK, a two-stage framework for targeted KV-memory repair. Stage 1 uses a disposable diagnostic adapter to identify harm-associated directions in multimodal key and value representations. Stage 2 projects out these directions, learns a lightweight residual repair, and anchors repaired keys with an image-structural prior to preserve visual grounding. Rather than applying the intervention uniformly, SPARK mixes repaired and original memory using a head-wise coefficient g_h* determined by intervention-relevant subspace energy E_h, requiring no explicit harm classifier at inference. Across LLaVA-OneVision-7B, Chameleon-7B, Qwen2-VL-7B, and InternVL2-4B, SPARK reduces multimodal attack success while preserving general capability. On LLaVA-OneVision-7B, image-only jailbreak attack success falls to 4.7%, while MMMU remains within 0.6 points of the undefended model (47.8 vs. 48.4) with near-baseline language quality. On MM-SafetyBench, attack success decreases from 39.2% to 12.4%. Even under white-box adaptive joint prompt-image attacks, attack success is limited to 20.3%, compared with 54.6% for the undefended model. These results suggest that multimodal jailbreak behavior can be substantially mitigated by selectively repairing intervention-relevant KV subspaces at prefill, particularly when harmful evidence is carried by the visual modality.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 28 references that are in the map. First posted on arXiv 13 Sep 2026.

Builds on

22 of 28 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Automating Steering for Safe Multimodal Large Language Models17 Jul
  2. 2025VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap14 Feb
  3. 2025Internal Activation Revision: Safeguarding Vision Language Models Without Parameter Update24 Jan
  4. 2024Cross-Modal Safety Mechanism Transfer in Large Vision-Language Models16 Oct
  5. 2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models11 Oct
  6. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov

Other references · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  7. 2022Mass-Editing Memory in a Transformercited by 976
  8. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  10. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  11. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  12. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
4 more
  1. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 78
  2. 2023ToViLaG: Your Visual-Language Generative Model is Also An Evildoercited by 27
  3. 2026Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamics Mechanisms and Efficient Alignmentcited by 4
  4. 2025Head-Specific Intervention Can Induce Misaligned AI Coordination in Large Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
  2. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models
  3. 2026GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models
  4. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts
  5. 2026SafeRI: Recognition and Intervention for Token-Level Safety Intervention in Large Vision Language Models
  6. 2025Dynamic Token Reweighting for Robust Vision-Language Models
  7. 2025DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Prompt
  8. 2026SGHA-Attack: Semantic-Guided Hierarchical Alignment for Transferable Targeted Attacks on Vision-Language Models