mlsanityData to 14 Oct 2026

Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospects

arXiv 2505.05318 ↗First posted 8 May 2025cs.CV cs.AI cs.CYcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The rapid adoption of Vision Language Models (VLMs), pre-trained on large image-text and video-text datasets, calls for protecting and informing users about when to trust these systems. This survey reviews studies on trust dynamics in user-VLM interactions, through a multi-disciplinary taxonomy encompassing different cognitive science capabilities, collaboration modes, and agent behaviours. Literature insights and findings from a workshop with prospective VLM users inform preliminary requirements for future VLM trust studies.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 68 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

48 of 68 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Trust but Verify: Programmatic VLM Evaluation in the Wild17 Oct
  2. 2024VHELM: A Holistic Evaluation of Vision Language Models9 Oct
  3. 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering16 Apr
  4. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning25 Mar
  5. 2023Adventures of Trustworthy Vision-Language Models: A Survey7 Dec
  6. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback16 Nov
  7. 2023Vision-Language Models for Vision Tasks: A Survey3 Apr
  8. 2022VLP: A Survey on Vision-Language Pre-training18 Feb

Other references · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Grounded Language-Image Pre-trainingcited by 1,156
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  6. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  7. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  8. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  9. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  10. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 251
  11. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  12. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
28 more
  1. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  2. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  3. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 135
  4. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 124
  5. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  6. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 120
  7. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 85
  8. 2024Can Large Language Models Reason and Plan?cited by 82
  9. 2024Safety of Multimodal Large Language Models on Images and Textscited by 75
  10. 2024Building Machines that Learn and Think with Peoplecited by 73
  11. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  12. 2024Conformal Alignment: Knowing When to Trust Foundation Models with Guaranteescited by 54
  13. 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examplescited by 53
  14. 2024ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videoscited by 51
  15. 2024Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Modelscited by 50
  16. 2023Vision-Language Interpreter for Robot Task Planningcited by 30
  17. 2017What Actions are Needed for Understanding Human Actions in Videos?cited by 29
  18. 2024ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Modelscited by 25
  19. 2024ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Capability for Large Vision-Language Modelscited by 25
  20. 2023Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Modelscited by 23
  21. 2022Unifying Vision-Language Representation Space with Single-tower Transformercited by 22
  22. 2024Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion?cited by 22
  23. 2024Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verificationcited by 17
  24. 2024Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Promptscited by 15
  25. 2024Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decodingcited by 15
  26. 2024Malicious Path Manipulations via Exploitation of Representation Vulnerabilities of Vision-Language Navigation Systemscited by 11
  27. 2024Probabilistic Conceptual Explainers: Trustworthy Conceptual Explanations for Vision Foundation Modelscited by 6
  28. 2024Evaluating and Enhancing Trustworthiness of LLMs in Perception Taskscited by 4

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Trust in Vision-Language Models: Insights from a Participatory User Workshopcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Dataset for Dynamic Human Preferences for Vision Language Models
  2. 2025Through Their Eyes: User Perceptions on Sensitive Attribute Inference of Social Media Videos by Visual Language Models
  3. 2026DiaVLo: Diagnosing Behaviours of Vision-Language Models
  4. 2025Building Trustworthy Multimodal AI: A Review of Fairness, Transparency, and Ethics in Vision-Language Tasks
  5. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions
  6. 2024What matters when building vision-language models?
  7. 2026Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflows
  8. 2026Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models