mlsanityData to 14 Oct 2026

Selectively Answering Visual Questions

arXiv 2406.00980 ↗First posted 3 Jun 2024cs.CL cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recently, large multi-modal models (LMMs) have emerged with the capacity to perform vision tasks such as captioning and visual question answering (VQA) with unprecedented accuracy. Applications such as helping the blind or visually impaired have a critical need for precise answers. It is specially important for models to be well calibrated and be able to quantify their uncertainty in order to selectively decide when to answer and when to abstain or ask for clarifications. We perform the first in-depth analysis of calibration methods and metrics for VQA with in-context learning LMMs. Studying VQA on two answerability benchmarks, we show that the likelihood score of visually grounded models is better calibrated than in their text-only counterparts for in-context learning, where sampling based methods are generally superior, but no clear winner arises. We propose Avg BLEU, a calibration score combining the benefits of both sampling and likelihood methods across modalities.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 35 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

23 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023PaLI-3 Vision Language Models: Smaller, Faster, Stronger13 Oct

Other references · 22

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  8. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  11. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  12. 2018Generating Natural Language Adversarial Examplescited by 424
10 more
  1. 2020Calibration of Pre-trained Transformerscited by 315
  2. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  3. 2020Selective Question Answering under Domain Shiftcited by 238
  4. 2020Captioning Images Taken by People Who Are Blindcited by 110
  5. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 89
  6. 2021Semantic Answer Similarity for Evaluating Question Answering Modelscited by 47
  7. 2021Correlated Input-Dependent Label Noise in Large-Scale Image Classificationcited by 37
  8. 2023Benchmarking Large Language Model Capabilities for Conditional Generationcited by 29
  9. 2023Improving Selective Visual Question Answering by Learning from Your Peerscited by 25
  10. 2023VQA Therapy: Exploring Answer Differences by Visually Grounding Answerscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Refine and Align: Confidence Calibration through Multi-Agent Interaction in VQAcited by 4
  2. 2026Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?cited by 2
  3. 2025Human Uncertainty-Aware Data Selection and Automatic Labeling in Visual Question Answeringcited by 1
  4. 2026Uncertainty-Aware Decision Making in Multimodal Large Language Modelscited by 0
  5. 2026Answer-Level Trust Selection for Physical Vision-Language Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types
  2. 2025Exploring Advanced Techniques for Visual Question Answering: A Comprehensive Comparison
  3. 2025Visual question answering: from early developments to recent advances -- a survey
  4. 2025Evaluating Variance in Visual Question Answering Benchmarks
  5. 2025Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models
  6. 2024Right this way: Can VLMs Guide Us to See More to Answer Questions?
  7. 2024DARE: Diverse Visual Question Answering with Robustness Evaluation
  8. 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering