ReFrame: Evidence-Guided Test-Time Safety Alignment in Multimodal Large Language Models
On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness
also in 2 more
Abstract (arXiv)
While multimodal large language models (MLLMs) extend model capabilities beyond text, they also make safety alignment increasingly challenging. Multimodal safety alignment methods must address cross-modal jailbreaks, safety-awareness failures, and over-sensitive refusals. However, existing methods often rely on retraining or internal-state inspection, limiting their applicability to deployed closed-source MLLMs and motivating test-time safety alignment. We analyze this setting and identify two key obstacles, utility dominance and reasoning inertia, which cause models to overlook latent risks or follow malicious reasoning trajectories. Guided by these insights, we propose ReFrame, a training-free multimodal input reframing framework where two agents share a lightweight locally deployed MLLM: the evidence-generation agent constructs complementary risk and utility evidence, and the rewrite-and-routing agent converts it into a safe proxy prompt and image-routing decision before calling the downstream MLLM, without modifying it or accessing its internal information. Experiments across multiple MLLMs and benchmarks show that ReFrame improves jailbreak defense, safety awareness, and oversensitivity reduction while preserving multimodal utility.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 32 references that are in the map. First posted on arXiv 21 Aug 2026.
Builds on
22 of 32 references.
Closest ancestors · 10
Same direction, newest first
- 2026Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory16 Mar
- 2026Pragma-VL: Towards a Pragmatic Arbitration of Safety and Helpfulness in MLLMs28 Feb
- 2025SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization17 Nov
- 2025Automating Steering for Safe Multimodal Large Language Models17 Jul
- 2025AMIA: Automatic Masking and Joint Intention Analysis Makes LVLMs Robust Jailbreak Defenders30 May
- 2025Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs16 Feb
- 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations14 Feb
- 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment27 Nov
- 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
- 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
Other references · 12
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2023Qwen Technical Reportcited by 3,273
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
- 2024Jailbreak Large Vision-Language Models Through Multi-Modal Linkagecited by 53
- 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27
- 2025Think in Safety: Unveiling and Mitigating Safety Alignment Collapse in Multimodal Large Reasoning Modelcited by 18
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Reimagining Safety Alignment with An Image
- 2026OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequences
- 2025DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models
- 2026MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs
- 2026SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models
- 2025Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy
- 2025Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?
- 2026MOSAIC: Composable Safety Alignment with Modular Control Tokens