mlsanityData to 14 Oct 2026

Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly

arXiv 2204.13631 ↗First posted 28 Apr 2022cs.CVcited by 89

Not on the map. Why?

Cited by 34 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Machine learning has advanced dramatically, narrowing the accuracy gap to humans in multimodal tasks like visual question answering (VQA). However, while humans can say "I don't know" when they are uncertain (i.e., abstain from answering a question), such ability has been largely neglected in multimodal research, despite the importance of this problem to the usage of VQA in real settings. In this work, we promote a problem formulation for reliable VQA, where we prefer abstention over providing an incorrect answer. We first enable abstention capabilities for several VQA models, and analyze both their coverage, the portion of questions answered, and risk, the error on that portion. For that, we explore several abstention approaches. We find that although the best performing models achieve over 70% accuracy on the VQA v2 dataset, introducing the option to abstain by directly using a model's softmax scores limits them to answering less than 7.5% of the questions to achieve a low risk of error (i.e., 1%). This motivates us to utilize a multimodal selection function to directly estimate the correctness of the predicted answers, which we show can increase the coverage by, for example, 2.3x from 6.8% to 15.6% at 1% risk. While it is important to analyze both coverage and risk, these metrics have a trade-off which makes comparing VQA models challenging. To address this, we also propose an Effective Reliability metric for VQA that places a larger cost on incorrect answers compared to abstentions. This new problem formulation, metric, and analysis for VQA provide the groundwork for building effective and reliable VQA models that have the self-awareness to abstain if and only if they don't know the answer.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Builds on 59 of its 90 references that are in the map. First posted on arXiv 28 Apr 2022.

Builds on

59 of 90 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2015VQA: Visual Question Answeringcited by 2,506
  11. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
38 more
  1. 2019Towards VQA Models That Can Readcited by 1,718
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  4. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  7. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  8. 2017Selective Classification for Deep Neural Networkscited by 852
  9. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  10. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 366
  11. 2020Calibrating Deep Neural Networks using Focal Losscited by 357
  12. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
  13. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 339
  14. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  15. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  16. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
  17. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  18. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  19. 2020Selective Question Answering under Domain Shiftcited by 238
  20. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  21. 2018To Trust Or Not To Trust A Classifiercited by 207
  22. 2019Addressing Failure Prediction by Learning Model Confidencecited by 186
  23. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  24. 2016Graph-Structured Representations for Visual Question Answeringcited by 123
  25. 2021Predicting with Confidence on Unseen Distributionscited by 116
  26. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 114
  27. 2019Cycle-Consistency for Robust Visual Question Answeringcited by 105
  28. 2017An Analysis of Visual Question Answering Algorithmscited by 90
  29. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  30. 2017Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challengecited by 76
  31. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  32. 2018Pythia v0.1: the Winning Entry to the VQA Challenge 2018cited by 62
  33. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  34. 2016Visual Question Answering: Datasets, Algorithms, and Future Challengescited by 56
  35. 2017IDK Cascades: Fast Deep Learning by Learning not to Overthinkcited by 48
  36. 2019Why Does a Visual Question Have Different Answers?cited by 34
  37. 2018Confidence Modeling for Neural Semantic Parsingcited by 33
  38. 2020Assessing Image Quality Issues for Real-World Problemscited by 28

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 14; 2024: 27; 2025: 18; 2026: 29.

89 papers.

Sort:

By area:

  1. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  2. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  3. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 105
  4. 2024SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMscited by 67
  5. 2024Conformal Alignment: Knowing When to Trust Foundation Models with Guaranteescited by 54
  6. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 41
  7. 2022Calibrated Interpretation: Confidence Estimation in Semantic Parsingcited by 33
  8. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 33
  9. 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answeringcited by 32
  10. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
  11. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 27
  12. 2023Improving Selective Visual Question Answering by Learning from Your Peerscited by 25
38 more
  1. 2023UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Modelscited by 25
  2. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  3. 2023HRVQA: A Visual Question Answering Benchmark for High-Resolution Aerial Imagescited by 21
  4. 2024Long-Form Answers to Visual Questions from Blind and Low Vision Peoplecited by 18
  5. 2023Model-Agnostic Gender Debiased Image Captioningcited by 17
  6. 2024TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoringcited by 17
  7. 2023Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Modelscited by 16
  8. 2024Enhancing Visual Question Answering through Ranking-Based Hybrid Training and Multimodal Fusioncited by 16
  9. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  10. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 14
  11. 2024Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Modelscited by 13
  12. 2023Exploring Question Decomposition for Zero-Shot VQAcited by 12
  13. 2024Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcementcited by 12
  14. 2023Simple Token-Level Confidence Improves Caption Correctnesscited by 11
  15. 2024Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix Itcited by 11
  16. 2024Overview of the EHRSQL 2024 Shared Task on Reliable Text-to-SQL Modeling on Electronic Health Recordscited by 11
  17. 2025Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questionscited by 11
  18. 2025MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questionscited by 11
  19. 2024Preemptive Detection and Correction of Misaligned Actions in LLM Agentscited by 10
  20. 2025Training-Free Personalization via Retrieval and Reasoning on Fingerprintscited by 10
  21. 2025When Robots Should Say "I Don't Know": Benchmarking Abstention in Embodied Question Answeringcited by 10
  22. 2026Visual Jev: Accurate and Efficient Decisions from Shared Visual Contextcited by 9
  23. 2024Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialoguescited by 8
  24. 2025Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probingcited by 8
  25. 2023Survey on Leveraging Uncertainty Estimation Towards Trustworthy Deep Neural Networks: The Case of Reject Option and Post-training Processingcited by 7
  26. 2023ProbVLM: Probabilistic Adapter for Frozen Vision-Language Modelscited by 7
  27. 2024Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparisoncited by 7
  28. 2025Do What? Teaching Vision-Language-Action Models to Reject the Impossiblecited by 7
  29. 2026When Silence Is Golden: Can LLMs Learn to Abstain in Temporal QA and Beyond?cited by 7
  30. 2026Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMscited by 7
  31. 2025Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Modelscited by 6
  32. 2023Did You Mean...? Confidence-based Trade-offs in Semantic Parsingcited by 5
  33. 2024Selectively Answering Visual Questionscited by 5
  34. 2025Variational Visual Question Answering for Uncertainty-Aware Selective Predictioncited by 5
  35. 2024Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Datacited by 4
  36. 2024Unanswerability Evaluation for Retrieval Augmented Generationcited by 4
  37. 2025Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensionalcited by 3
  38. 2025HARMONY: Hidden Activation Representations and Model Output-Aware Uncertainty Estimation for Vision-Language Modelscited by 3

50 most cited of 89 All 89 (JSON)