mlsanityData to 7 Oct 2026

Trust in Vision-Language Models: Insights from a Participatory User Workshop

arXiv 2511.13458 ↗First posted 17 Nov 2025cs.HC cs.AI cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

With the growing deployment of Vision-Language Models (VLMs), pre-trained on large image-text and video-text datasets, it is critical to equip users with the tools to discern when to trust these systems. However, examining how user trust in VLMs builds and evolves remains an open problem. This problem is exacerbated by the increasing reliance on AI models as judges for experimental validation, to bypass the cost and implications of running participatory design studies directly with users. Following a user-centred approach, this paper presents preliminary results from a workshop with prospective VLM users. Insights from this pilot workshop inform future studies aimed at contextualising trust metrics and strategies for participants' engagement to fit the case of user-VLM interaction.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 52 references that are in the map. First posted on arXiv 17 Nov 2025.

Builds on

30 of 52 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospects8 May
  2. 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering16 Apr
  3. 2023Adventures of Trustworthy Vision-Language Models: A Survey7 Dec
  4. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Models27 Nov
  5. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback16 Nov
  6. 2022VLP: A Survey on Vision-Language Pre-training18 Feb

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Grounded Language-Image Pre-trainingcited by 1,147
  3. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  6. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  7. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  8. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  9. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  10. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  11. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  12. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
12 more
  1. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 114
  2. 2024Can Large Language Models Reason and Plan?cited by 82
  3. 2024Building Machines that Learn and Think with Peoplecited by 73
  4. 2024Safety of Multimodal Large Language Models on Images and Textscited by 70
  5. 2024ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videoscited by 51
  6. 2024Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Modelscited by 48
  7. 2024ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Capability for Large Vision-Language Modelscited by 24
  8. 2024Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion?cited by 22
  9. 2022Unifying Vision-Language Representation Space with Single-tower Transformercited by 21
  10. 2024Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verificationcited by 17
  11. 2024Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decodingcited by 14
  12. 2024Malicious Path Manipulations via Exploitation of Representation Vulnerabilities of Vision-Language Navigation Systemscited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Through Their Eyes: User Perceptions on Sensitive Attribute Inference of Social Media Videos by Visual Language Models
  2. 2025Building Trustworthy Multimodal AI: A Review of Fairness, Transparency, and Ethics in Vision-Language Tasks
  3. 2026A Dataset for Dynamic Human Preferences for Vision Language Models
  4. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions
  5. 2026VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understanding
  6. 2026Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflows
  7. 2024Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment
  8. 2025To Trust Or Not To Trust Your Vision-Language Model's Prediction