mlsanityData to 7 Oct 2026

Do VLMs Have a Moral Backbone? A Study on the Fragile Morality of Vision-Language Models

arXiv 2601.17082 ↗First posted 23 Jan 2026cs.CY cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Despite substantial efforts toward improving the moral alignment of Vision-Language Models (VLMs), it remains unclear whether their ethical judgments are stable in realistic settings. This work studies moral robustness in VLMs, defined as the ability to preserve moral judgments under textual and visual perturbations that do not alter the underlying moral context. We systematically probe VLMs with a diverse set of model-agnostic multimodal perturbations and find that their moral stances are highly fragile, frequently flipping under simple manipulations. Our analysis reveals systematic vulnerabilities across perturbation types, moral domains, and model scales, including a sycophancy trade-off where stronger instruction-following models are more susceptible to persuasion. We further show that lightweight inference-time interventions can partially restore moral stability. These results demonstrate that moral alignment alone is insufficient and that moral robustness is a necessary criterion for the responsible deployment of VLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 37 references that are in the map. First posted on arXiv 23 Jan 2026.

Builds on

18 of 37 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025MORALISE: A Structured Benchmark for Moral Alignment in Visual Language Models20 May
  2. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?4 Mar
  3. 2023Vision-Language Models for Vision Tasks: A Survey3 Apr

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  4. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  7. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  8. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  9. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  10. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  11. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 78
  12. 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 51
3 more
  1. 2024Measuring and Benchmarking Large Language Models' Capabilities to Generate Persuasive Languagecited by 31
  2. 2024BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attackscited by 28
  3. 2024Adversarial Attacks of Vision Tasks in the Past 10 Years: A Surveycited by 22

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Moral Sycophancy in Vision Language Models
  2. 2024MM-MoralBench: A MultiModal Moral Evaluation Benchmark for Large Vision-Language Models
  3. 2026Visual Distraction Undermines Moral Reasoning in Vision-Language Models
  4. 2026MM-SCALE: Grounded Multimodal Moral Reasoning via Scalar Judgment and Listwise Alignment
  5. 2026The Fragility Of Moral Judgment In Large Language Models
  6. 2026Position: Evaluations of AI Moral Reasoning Still Miss Half of the Picture
  7. 2026Representational alignment yields generalizable safety in language models
  8. 2026Agreement Is Not Alignment: Divergent Moral Grounds in Human and LLM Ethical Judgments