mlsanityData to 7 Oct 2026

CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration

arXiv 2409.11365 ↗First posted 17 Sep 2024cs.CLcited by 30

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

The deployment of multimodal large language models (MLLMs) has demonstrated remarkable success in engaging in conversations involving visual inputs, thanks to the superior power of large language models (LLMs). Those MLLMs are typically built based on the LLMs, with an image encoder to process images into the token embedding space of the LLMs. However, the integration of visual modality has introduced a unique vulnerability: the MLLM becomes susceptible to malicious visual inputs and prone to generating sensitive or harmful responses, even though the LLM has been trained on textual dataset to align with human value. In this paper, we first raise the question: ``Do the MLLMs possess safety-awareness against malicious image inputs?". We find that after adding a principle that specifies the safety requirement into the input of the MLLM, the model's safety awareness becomes boosted. This phenomenon verifies the existence of MLLM's safety-awareness against image inputs, it is only weakened by the modality gap. We then introduce a simple yet effective technique termed CoCA, which amplifies the safety-awareness of the MLLM by calibrating its output distribution. Our proposed strategy helps the model reclaim its original safety awareness without losing its original capabilities. We verify the effectiveness of our approach on both multimodal safety and understanding benchmarks.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 89 references that are in the map. First posted on arXiv 17 Sep 2024.

Builds on

42 of 89 references.

References · 42

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  11. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  12. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
30 more
  1. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  2. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  4. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  5. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  6. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  7. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  8. 2018AI safety via debatecited by 384
  9. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  10. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  11. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  12. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 284
  13. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  14. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  15. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  16. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  17. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  18. 2018Supervising strong learners by amplifying weak expertscited by 139
  19. 2023Training Language Models with Language Feedback at Scalecited by 125
  20. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 87
  21. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
  22. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  23. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  24. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 48
  25. 2024Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptionscited by 43
  26. 2024The Instinctive Bias: Spurious Images lead to Illusion in MLLMscited by 11
  27. 2024Jailbreaking as a Reward Misspecification Problemcited by 11
  28. 2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10
  29. 2020UNISON: Unpaired Cross-lingual Image Captioningcited by 8
  30. 2023Improving Language Model Integration for Neural Machine Translationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 19; 2026: 8.

30 papers.

Sort:

By area:

  1. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  2. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  3. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 34
  4. 2025Towards Trustworthy GUI Agents: A Surveycited by 33
  5. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defensecited by 20
  6. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  7. 2025Automating Steering for Safe Multimodal Large Language Modelscited by 17
  8. 2025Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Modelscited by 13
  9. 2025SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimizationcited by 11
  10. 2025VLMGuard-R1: Proactive Safety Alignment for VLMs via Reasoning-Driven Prompt Optimizationcited by 10
  11. 2025OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluationcited by 10
  12. 2025Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?cited by 9
18 more
  1. 2025SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanismcited by 9
  2. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  3. 2025SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Modelscited by 8
  4. 2025EcoAlign: An Economically Rational Framework for Efficient LVLM Alignmentcited by 5
  5. 2025Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignmentcited by 4
  6. 2025Bootstrapping LLM Robustness for VLM Safety via Reducing the Pretraining Modality Gapcited by 3
  7. 2025A Survey on Training-free Alignment of Large Language Modelscited by 2
  8. 2026Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utilitycited by 2
  9. 2026Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Modelscited by 2
  10. 2026Safe responses matter: Output-aware safety guardrail mitigate over-refusal in MLLMscited by 2
  11. 2025Dynamic Token Reweighting for Robust Vision-Language Modelscited by 1
  12. 2025SGM: Safety Glasses for Multimodal Large Language Models via Neuron-Level Detoxificationcited by 1
  13. 2026GuardAlign: Test-time Safety Alignment in Multimodal Large Language Modelscited by 1
  14. 2024CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Modelcited by 0
  15. 2026Safety-Potential Pruning for Enhancing Safety Prompts Against VLM Jailbreaking Without Retrainingcited by 0
  16. 2026COMIC: Reference-Aware Safety Gating for Multimodal Large Language Modelscited by 0
  17. 2026Transfer Safety Awareness for Cross-Modal Safety Drift in Multimodal Large Language Modelscited by 0
  18. 2026SafeRI: Recognition and Intervention for Token-Level Safety Intervention in Large Vision Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs
  2. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation
  3. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance
  4. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models
  5. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models
  6. 2024Safety of Multimodal Large Language Models on Images and Texts
  7. 2025Steering Multimodal Large Language Models Decoding for Context-Aware Safety
  8. 2026SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models