mlsanityData to 7 Oct 2026

Images Amplify Misinformation Sharing in Vision-Language Models

arXiv 2505.13302 ↗First posted 19 May 2025cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

As language and vision-language models (VLMs) become central to information access and online interaction, concerns grow about their potential to amplify misinformation. Human studies show that images boost the perceived credibility and shareability of information, raising the question of whether VLMs exhibit the same vulnerability. We present the first study examining how images influence VLMs' propensity to reshare news content, how this effect varies across model families, and how persona conditioning and content attributes modulate such behavior. We develop a jailbreaking-inspired prompting strategy that bypasses VLMs' default refusals to engage with controversial news, allowing them to generate resharing decisions across diverse topics and elicited traits, including antisocial ones. We evaluate four state-of-the-art VLMs on a novel multimodal dataset of fact-checked political news from PolitiFact, paired with images and ground-truth veracity labels. Our experiments show that image presence increases resharing rates by 14.5% for false news and 5.3% for true news. Persona conditioning further modulates this effect: Dark Triad traits amplify resharing of false news, whereas Republican-aligned profiles reduce sensitivity to veracity. Among the tested models, Claude-3-Haiku demonstrates the greatest robustness to visual misinformation. These findings reveal that VLMs replicate human-like biases in response to images, underscoring emerging risks for multimodal AI systems. They point to the need for evaluation frameworks and mitigation strategies that account for visual influence and persona-driven variability, particularly in sociotechnical settings where AI systems shape public discourse and information sharing.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 88 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

19 of 88 references.

References · 19

  1. 2023Whose Opinions Do Language Models Reflect?cited by 759
  2. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  3. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  4. 2018FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Mediacited by 281
  5. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 172
  6. 2024Quantifying the Persona Effect in LLM Simulationscited by 164
  7. 2023Generative Language Models Exhibit Social Identity Biasescited by 107
  8. 2022End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Modelscited by 105
  9. 2025LLM Generated Persona is a Promise with a Catchcited by 77
  10. 2023AI model GPT-3 (dis)informs us better than humanscited by 75
  11. 2024SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detectioncited by 73
  12. 2024Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometricscited by 56
7 more
  1. 2019r/Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detectioncited by 53
  2. 2024Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personascited by 36
  3. 2024Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personascited by 23
  4. 2024Large Language Model-driven Multi-Agent Simulation for News Diffusion Under Different Network Structurescited by 14
  5. 2024Comparing the willingness to share for human-generated vs. AI-generated fake newscited by 12
  6. 2024Behind the Screen: Investigating ChatGPT's Dark Personality Traits and Conspiracy Beliefscited by 2
  7. 2025Can dialogues with AI systems help humans better discern visual misinformation?cited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Brand-as-Memory: Vision-Language Models Encode Causal, Mechanistically Localizable Credibility Priors for News Sources
  2. 2025Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessment
  3. 2025Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Models
  4. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs
  5. 2025Through Their Eyes: User Perceptions on Sensitive Attribute Inference of Social Media Videos by Visual Language Models
  6. 2025Do Vision-Language Models Understand Visual Persuasiveness? A Diagnosis via Visual Persuasive Factors
  7. 2026Seeing Red, Thinking Bad: Color Bias in Vision Language Models
  8. 2025VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models