mlsanityData to 14 Oct 2026

Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning

arXiv 2402.15610 ↗First posted 23 Feb 2024cs.CLcited by 40

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Selective prediction minimizes incorrect predictions from vision-language models (VLMs) by allowing them to abstain from answering when uncertain. However, when deploying a vision-language system with low tolerance for inaccurate predictions, selective prediction may be over-cautious and abstain too frequently, even on many correct predictions. We introduce ReCoVERR, an inference-time algorithm to reduce the over-abstention of a selective vision-language system without increasing the error rate of the system's predictions. When the VLM makes a low-confidence prediction, instead of abstaining ReCoVERR tries to find relevant clues in the image that provide additional evidence for the prediction. ReCoVERR uses an LLM to pose related questions to the VLM, collects high-confidence evidences, and if enough evidence confirms the prediction the system makes a prediction instead of abstaining. ReCoVERR enables three VLMs (BLIP2, InstructBLIP, and LLaVA-1.5) to answer up to 20% more questions on the VQAv2 and A-OKVQA tasks without decreasing system accuracy, thus improving overall system reliability. Our code is available at https://github.com/tejas1995/ReCoVERR.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 15 of its 43 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

15 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models9 Oct
  2. 2023IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models24 May
  3. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar

Other references · 12

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2023Mistral 7Bcited by 3,164
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  5. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  6. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  7. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  8. 2023Modular Visual Question Answering via Code Generationcited by 48
  9. 2020Towards Ecologically Valid Research on Language User Interfacescited by 42
  10. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 41
  11. 2023Improving Automatic VQA Evaluation Using Large Language Modelscited by 37
  12. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 8; 2026: 27.

40 papers.

Sort:

By area:

  1. 2024Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinationscited by 54
  2. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 14
  3. 2025Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questionscited by 11
  4. 2024When to Speak, When to Abstain: Contrastive Decoding with Abstentioncited by 8
  5. 2024Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparisoncited by 7
  6. 2025KnowGuard: Knowledge-Driven Abstention for Multi-Round Clinical Reasoningcited by 7
  7. 2025Know What You do Not Know: Verbalized Uncertainty Estimation Robustness on Corrupted Images in Vision-Language Modelscited by 6
  8. 2025Variational Visual Question Answering for Uncertainty-Aware Selective Predictioncited by 5
  9. 2026The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Modelscited by 5
  10. 2024Transparent and Coherent Procedural Mistake Detectioncited by 4
  11. 2025SafeCoT: Improving VLM Safety with Minimal Reasoningcited by 3
  12. 2025Can VLMs Recall Factual Associations From Visual References?cited by 3
28 more
  1. 2025HARMONY: Hidden Activation Representations and Model Output-Aware Uncertainty Estimation for Vision-Language Modelscited by 3
  2. 2026VLM-UQBench: A Benchmark for Modality-Specific and Cross-Modality Uncertainties in Vision Language Modelscited by 3
  3. 2026Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Modelcited by 3
  4. 2026Knowing When to Answer: Adaptive Confidence Refinement for Reliable Audio-Visual Question Answeringcited by 2
  5. 2026Leveraging Data to Say No: Memory Augmented Plug-and-Play Selective Predictioncited by 1
  6. 2026Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarkscited by 1
  7. 2026Geometric Risk Control for Vision-Language Model OCRcited by 1
  8. 2026SIEVES: Selective Prediction Generalizes through Visual Evidence Scoringcited by 1
  9. 2026AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficultycited by 1
  10. 2025Improving VQA Reliability: A Dual-Assessment Approach with Self-Reflection and Cross-Model Verificationcited by 0
  11. 2026Learning When Not to Decide: A Framework for Overcoming Factual Presumptuousness in AI Adjudicationcited by 0
  12. 2026A Regime Theory of Controller Class Selection for LLM Action Decisionscited by 0
  13. 2026Uncertainty Quantification for Multimodal Retrieval Augmented Generationcited by 0
  14. 2026It Takes One to Bias Them All: Breaking Bad with One-Shot GRPOcited by 0
  15. 2026When Depth Is Better Told Than Shown: Depth-Ordinal Prompting for Vision-Language Spatial Reasoningcited by 0
  16. 2026When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Renderingcited by 0
  17. 2026Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Readingcited by 0
  18. 2026How Do VLMs Behave When Blind or Misled? Behavioral Evaluation of VLMs on Scientific Figurescited by 0
  19. 2026Uncertainty-Aware Decision Making in Multimodal Large Language Modelscited by 0
  20. 2026SURE-Voice: A Front-End Baseline for Speech-Evidence Filtering in Speech LLMscited by 0
  21. 2026Target-Checked Reliability Score Refinement for Video Question Answeringcited by 0
  22. 2026Symmetry-Aware Likelihood-Orbit Aggregation for Selective Left-Right Claim Verificationcited by 0
  23. 2026RECAP: Relation Evidence Calibration for Detecting Spatial Relation Hallucinations in Vision-Language Modelscited by 0
  24. 2026ViCoR: Reliable Molecular Structure Extraction via Spatially Aligned Verification and Executable Revisioncited by 0
  25. 2026When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Promptscited by 0
  26. 2026Strong yet Controllable LLM Abstention without Learning to Abstaincited by 0
  27. 2026Lost in Contrast: Instruction-Preserving Decoding for Reliable Visual Question Answeringcited by 0
  28. 2026Visual Dependence Does Not Imply Answerability: A Systematic Analysis of Answerability Signals in Frozen InstructBLIPcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering
  2. 2026Answer-Level Trust Selection for Physical Vision-Language Reasoning
  3. 2026Dual-Latent Memory Routing for Vision-Language Reasoning
  4. 2026Teaching Vision-Language Models to Decline: Support Completion Against Mirage
  5. 2025VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training
  6. 2024SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information
  7. 2025Cache-of-Thought: Master-Apprentice Framework for Cost-Effective Vision Language Model Reasoning
  8. 2025Unbiased Visual Reasoning with Controlled Visual Inputs