mlsanityData to 7 Oct 2026

Improving Automatic VQA Evaluation Using Large Language Models

arXiv 2310.02567 ↗First posted 4 Oct 2023cs.CV cs.AI cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

8 years after the visual question answering (VQA) task was proposed, accuracy remains the primary metric for automatic evaluation. VQA Accuracy has been effective so far in the IID evaluation setting. However, our community is undergoing a shift towards open-ended generative models and OOD evaluation. In this new paradigm, the existing VQA Accuracy metric is overly stringent and underestimates the performance of VQA systems. Thus, there is a need to develop more robust automatic VQA metrics that serve as a proxy for human judgment. In this work, we propose to leverage the in-context learning capabilities of instruction-tuned large language models (LLMs) to build a better VQA metric. We formulate VQA evaluation as an answer-rating task where the LLM is instructed to score the accuracy of a candidate answer given a set of reference answers. We demonstrate the proposed metric better correlates with human judgment compared to existing metrics across several VQA models and benchmarks. We hope wide adoption of our metric will contribute to better estimating the research progress on the VQA task. We plan to release the evaluation code and collected human judgments.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

28 of 38 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
16 more
  1. 2015VQA: Visual Question Answeringcited by 2,489
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  5. 2023LIMA: Less Is More for Alignmentcited by 1,080
  6. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  7. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  8. 2023GPTScore: Evaluate as You Desirecited by 357
  9. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  10. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 110
  11. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  12. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  13. 2021Semantic Answer Similarity for Evaluating Question Answering Modelscited by 47
  14. 2021What's in a Name? Answer Equivalence For Open-Domain Question Answeringcited by 21
  15. 2022Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalizationcited by 20
  16. 2021'Just because you are right, doesn't mean I am wrong': Overcoming a Bottleneck in the Development and Evaluation of Open-Ended Visual Question Answering (VQA) Taskscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 18; 2026: 8.

37 papers.

Sort:

By area:

  1. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  2. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  3. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  4. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
  5. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  6. 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoningcited by 37
  7. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 37
  8. 2024Biomedical Visual Instruction Tuning with Clinician Preference Alignmentcited by 21
  9. 2024Long-Form Answers to Visual Questions from Blind and Low Vision Peoplecited by 18
  10. 2025Is Your Video Language Model a Reliable Judge?cited by 14
  11. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 13
  12. 2025MSEarth: A Multimodal Benchmark for Earth Science Phenomenon Discovery with MLLMscited by 9
25 more
  1. 2025MultiFinRAG: An Optimized Multimodal Retrieval-Augmented Generation (RAG) Framework for Financial Question Answeringcited by 9
  2. 2025CLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answeringcited by 6
  3. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?cited by 6
  4. 2024Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Modelcited by 5
  5. 2024ERVQA: A Dataset to Benchmark the Readiness of Large Vision Language Models in Hospital Environmentscited by 5
  6. 2025Describe Anything Model for Visual Question Answering on Text-rich Imagescited by 5
  7. 2026EarthVL: A Progressive Earth Vision-Language Understanding and Generation Frameworkcited by 5
  8. 2025UWBench: A Comprehensive Vision-Language Benchmark for Underwater Understandingcited by 4
  9. 2026Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Modelscited by 4
  10. 2025HARMONY: Hidden Activation Representations and Model Output-Aware Uncertainty Estimation for Vision-Language Modelscited by 3
  11. 2025Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?cited by 2
  12. 2024SimpsonsVQA: Enhancing Inquiry-Based Learning with a Tailored Datasetcited by 1
  13. 2025All-in-one: Understanding and Generation in Multimodal Reasoning with the MAIA Benchmarkcited by 1
  14. 2025Adapting Vision-Language Models for Evaluating World Modelscited by 1
  15. 2025SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluationcited by 1
  16. 2026Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guaranteescited by 1
  17. 2025Patent Figure Classification using Large Vision-language Modelscited by 0
  18. 2025EmoAssist: Emotional Assistant for Visual Impairment Communitycited by 0
  19. 2025SweeperBot: Making 3D Browsing Accessible through View Analysis and Visual Question Answeringcited by 0
  20. 2025LM4Opt-RA: A Multi-Candidate LLM Framework with Structured Ranking for Automating Network Resource Allocationcited by 0
  21. 2026CoTZero: Annotation-Free Human-Like Vision Reasoning via Hierarchical Synthetic CoTcited by 0
  22. 2026Phantasia: Context-Adaptive Backdoors in Vision Language Modelscited by 0
  23. 2026RADIANCE: Relative Adaptive Denoising with IP-Adapter for Novel Concept Enhancementcited by 0
  24. 2026How Much Does It Cost to Answer My Question? Benchmarking Cloud VLM-based VQA Systemscited by 0
  25. 2026RA-CoA: Training-free Fashion Image Captioning via Retrieval-Augmented Chain-of-Attributescited by 0