mlsanityData to 7 Oct 2026

GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models

arXiv 2602.24027 ↗First posted 27 Feb 2026cs.CV cs.MMcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 2 more
Abstract (arXiv)

Large vision-language models (LVLMs) have achieved remarkable progress in vision-language reasoning tasks, yet ensuring their safety remains a critical challenge. Recent input-side defenses detect unsafe images with CLIP and prepend safety prefixes to prompts, but they still suffer from inaccurate detection in complex scenes and unstable safety signals during decoding. To address these issues, we propose GuardAlign, a training-free defense framework that integrates two strategies. First, OT-enhanced safety detection leverages optimal transport to measure distribution distances between image patches and unsafe semantics, enabling accurate identification of malicious regions without additional computational cost. Second, cross-modal attentive calibration strengthens the influence of safety prefixes by adaptively reallocating attention across layers, ensuring that safety signals remain consistently activated throughout generation. Extensive evaluations on six representative MLLMs demonstrate that GuardAlign reduces unsafe response rates by up to 39% on SPA-VL, while preserving utility, achieving an improvement on VQAv2 from 78.51% to 79.21%.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 64 references that are in the map. First posted on arXiv 27 Feb 2026.

Builds on

58 of 64 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025AMIA: Automatic Masking and Joint Intention Analysis Makes LVLMs Robust Jailbreak Defenders30 May
  2. 2025Dynamic Token Reweighting for Robust Vision-Language Models22 May
  3. 2025VLMGuard-R1: Proactive Safety Alignment for VLMs via Reasoning-Driven Prompt Optimization17 Apr
  4. 2025Understanding and Rectifying Safety Perception Distortion in VLMs18 Feb
  5. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time9 Oct
  6. 2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration17 Sep
  7. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models26 Jun
  8. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model17 Jun
  9. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
  10. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting14 Mar
  11. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance5 Jan
  12. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov
2 more
  1. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  2. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
32 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  6. 2018Computational Optimal Transportcited by 765
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  9. 2015Optimal Transport for Domain Adaptationcited by 457
  10. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  11. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  12. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  13. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  14. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 195
  15. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  16. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  17. 2019Scalable Gromov-Wasserstein Learning for Graph Partitioning and Matchingcited by 102
  18. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  19. 2020Iterative label cleaning for transductive and semi-supervised few-shot learningcited by 40
  20. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  21. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  22. 2023Tuning Multi-mode Token-level Prompt Alignment across Modalitiescited by 26
  23. 2025Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimizationcited by 22
  24. 2025AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Modelscited by 15
  25. 2022Adaptive Distribution Calibration for Few-Shot Learning with Hierarchical Optimal Transportcited by 14
  26. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 14
  27. 2024Self-Evaluation as a Defense Against Adversarial Attacks on LLMscited by 13
  28. 2025Dynamic Multimodal Prototype Learning in Vision-Language Modelscited by 13
  29. 2025Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoningcited by 12
  30. 2024Video Repurposing from User Generated Content: A Large-scale Dataset and Benchmarkcited by 9
  31. 2025Hierarchical Semantic Alignment for Image Clusteringcited by 5
  32. 2021Few-shot Partial Multi-view Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Modelscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025VisuoAlign: Safety Alignment of LVLMs with Multimodal Tree Search
  2. 2025VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap
  3. 2025DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Prompt
  4. 2026SafeRI: Recognition and Intervention for Token-Level Safety Intervention in Large Vision Language Models
  5. 2025MSR-Align: Policy-Grounded Multimodal Alignment for Safety-Aware Reasoning in Vision-Language Models
  6. 2025Think-Reflect-Revise: A Policy-Guided Reflective Framework for Safety Alignment in Large Vision Language Models
  7. 2025Reimagining Safety Alignment with An Image
  8. 2026Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility