mlsanityData to 7 Oct 2026

Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images

arXiv 2603.08486 ↗First posted 9 Mar 2026cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 2 more
Abstract (arXiv)

Multimodal large language models (MLLMs) face safety misalignment, where visual inputs enable harmful outputs. To address this, existing methods require explicit safety labels or contrastive data; yet, threat-related concepts are concrete and visually depictable, while safety concepts, like helpfulness, are abstract and lack visual referents. Inspired by the Self-Fulfilling mechanism underlying emergent misalignment, we propose Visual Self-Fulfilling Alignment (VSFA). VSFA fine-tunes vision-language models (VLMs) on neutral VQA tasks constructed around threat-related images, without any safety labels. Through repeated exposure to threat-related visual content, models internalize the implicit semantics of vigilance and caution, shaping safety-oriented personas. Experiments across multiple VLMs and safety benchmarks demonstrate that VSFA reduces the attack success rate, improves response quality, and mitigates over-refusal while preserving general capabilities. Our work extends the self-fulfilling mechanism from text to visual modalities, offering a label-free approach to VLMs alignment.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 83 references that are in the map. First posted on arXiv 9 Mar 2026.

Builds on

44 of 83 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Models12 Oct
  2. 2025The Safety Reminder: A Soft Prompt to Reactivate Delayed Safety Awareness in Vision-Language Models15 Jun
  3. 2025Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning14 Mar
  4. 2025Understanding and Rectifying Safety Perception Distortion in VLMs18 Feb
  5. 2025VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap14 Feb
  6. 2024RapGuard: Safeguarding Multimodal Large Language Models via Rationale-aware Defensive Prompting25 Dec
  7. 2024PBI-Attack: Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for Toxicity Maximization8 Dec
  8. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safety29 Nov
  9. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting14 Mar
  10. 2024Safety of Multimodal Large Language Models on Images and Texts1 Feb
  11. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2025Qwen3-VL Technical Reportcited by 2,534
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  8. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  9. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  10. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 133
  11. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
  12. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 89
21 more
  1. 2025Persona Features Control Emergent Misalignmentcited by 76
  2. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 70
  3. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  4. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 54
  5. 2025When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Modelscited by 46
  6. 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
  7. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
  8. 2025Understanding the Repeat Curse in Large Language Models from a Feature Perspectivecited by 36
  9. 2024Exploring the Personality Traits of LLMs through Latent Features Steeringcited by 21
  10. 2025Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMscited by 20
  11. 2025Model-agnostic Adversarial Attack and Defense for Vision-Language-Action Modelscited by 19
  12. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 18
  13. 2025Goal-oriented Backdoor Attack against Vision-Language-Action Models via Physical Objectscited by 16
  14. 2024Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustnesscited by 13
  15. 2025Understanding and Mitigating Cross-lingual Privacy Leakage via Language-specific and Universal Privacy Neuronscited by 13
  16. 2026StackPlanner: A Centralized Hierarchical Multi-Agent System with Task-Experience Memory Managementcited by 13
  17. 2024Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMscited by 11
  18. 2025PIXEL: Adaptive Steering Via Position-wise Injection with eXact Estimated Levels under Subspace Calibrationcited by 10
  19. 2025OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluationcited by 10
  20. 2025Re-Emergent Misalignment: How Narrow Fine-Tuning Erodes Safety Alignment in LLMscited by 9
  21. 2024From Concrete to Abstract: A Multimodal Generative Approach to Abstract Concept Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Multi-Adapter Representation Interventions via Energy Calibrationcited by 0
  2. 2026The Uncontrolled Variable: Vision-Language Model Refusal Responds to Image Presence in Ways Risk Cannot Explaincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026When Seeing Overrides Knowing: Visual Dominance and Deferral-Based Method for Personalized Safety in VLMs
  2. 2025Security Tensors as a Cross-Modal Bridge: Extending Text-Aligned Safety to Vision in LVLM
  3. 2025Self-Aware Safety Augmentation: Leveraging Internal Semantic Understanding to Enhance Safety in Vision-Language Models
  4. 2025Think-Reflect-Revise: A Policy-Guided Reflective Framework for Safety Alignment in Large Vision Language Models
  5. 2025VisuoAlign: Safety Alignment of LVLMs with Multimodal Tree Search
  6. 2025Reimagining Safety Alignment with An Image
  7. 2025SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization
  8. 2026Jailbreaking Vision-Language Models Through the Visual Modality