mlsanityData to 7 Oct 2026

Trust but Verify: Programmatic VLM Evaluation in the Wild

arXiv 2410.13121 ↗First posted 17 Oct 2024cs.CV cs.AIcited by 5

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Models (VLMs) often generate plausible but incorrect responses to visual queries. However, reliably quantifying the effect of such hallucinations in free-form responses to open-ended queries is challenging as it requires visually verifying each claim within the response. We propose Programmatic VLM Evaluation (PROVE), a new benchmarking paradigm for evaluating VLM responses to open-ended queries. To construct PROVE, we provide a large language model (LLM) with a high-fidelity scene-graph representation constructed from a hyper-detailed image caption, and prompt it to generate diverse question-answer (QA) pairs, as well as programs that can be executed over the scene graph object to verify each QA pair. We thus construct a benchmark of 10.5k challenging but visually grounded QA pairs. Next, to evaluate free-form model responses to queries in PROVE, we propose a programmatic evaluation strategy that measures both the helpfulness and truthfulness of a response within a unified scene graph-based framework. We benchmark the helpfulness-truthfulness trade-offs of a range of VLMs on PROVE, finding that very few are in-fact able to achieve a good balance between the two. Project page: urlhttps://prove-explorer.netlify.app/.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 53 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

39 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences16 Jun

Other references · 38

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
26 more
  1. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  2. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  5. 2018Object Hallucination in Image Captioningcited by 661
  6. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  7. 2023Segment Everything Everywhere All at Oncecited by 546
  8. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  9. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  10. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  11. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  12. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  13. 2024The Platonic Representation Hypothesiscited by 362
  14. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  15. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  16. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  17. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254
  18. 2019Connecting Vision and Language with Localized Narrativescited by 225
  19. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  20. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  21. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Modelscited by 99
  22. 2024LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBenchcited by 93
  23. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  24. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuningcited by 69
  25. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  26. 2024FIHA: Autonomous Hallucination Evaluation in Vision-Language Models with Davidson Scene Graphscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025VLM-SlideEval: Evaluating VLMs on Structured Comprehension and Perturbation Sensitivity in PPTcited by 2
  2. 2025Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospectscited by 1
  3. 2026SciFigAlign: Scoring Scientific Figures by Fine-tuned Alignment of Visuals with Manuscript Evidencecited by 0
  4. 2026Physics as the label for measuring and correcting materials reasoning in multimodal modelscited by 0
  5. 2026Program-Verified Self-Evolution for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questions
  2. 2024TUBench: Benchmarking Large Vision-Language Models on Trustworthiness with Unanswerable Questions
  3. 2026Composition, Not Conversation: VLMs Lose the Scene, Not the Thread
  4. 2026When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias
  5. 2026SpatialChain: A Benchmark for Auditing Spatial Reasoning Faithfulness in VLMs
  6. 2025REVAL: A Comprehension Evaluation on Reliability and Values of Large Vision-Language Models
  7. 2026VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence
  8. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark