# Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning

arXiv: https://arxiv.org/abs/2402.15610 · HTML: https://mlsanity.com/paper/2402.15610/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 15 of its 43 references that are in the map.
- First posted on arXiv 23 Feb 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 15 of 15)

- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents](https://mlsanity.com/paper/2306.16527/) (2023)
- [Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2](https://mlsanity.com/paper/2311.10702/) (2023)
- [Modular Visual Question Answering via Code Generation](https://mlsanity.com/paper/2306.05392/) (2023)
- [IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models](https://mlsanity.com/paper/2305.14985/) (2023)
- [Towards Ecologically Valid Research on Language User Interfaces](https://mlsanity.com/paper/2007.14435/) (2020)
- [See, Say, and Segment: Teaching LMMs to Overcome False Premises](https://mlsanity.com/paper/2312.08366/) (2023)
- [Improving Automatic VQA Evaluation Using Large Language Models](https://mlsanity.com/paper/2310.02567/) (2023)
- [Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning](https://mlsanity.com/paper/2310.00647/) (2023)
- [Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models](https://mlsanity.com/paper/2310.05861/) (2023)

## Built on by (most cited first, 30 of 40)

- [Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations](https://mlsanity.com/paper/2404.10960/) (2024)
- [Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness](https://mlsanity.com/paper/2407.01942/) (2024)
- [Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions](https://mlsanity.com/paper/2507.13773/) (2025)
- [When to Speak, When to Abstain: Contrastive Decoding with Abstention](https://mlsanity.com/paper/2412.12527/) (2024)
- [Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparison](https://mlsanity.com/paper/2407.07840/) (2024)
- [KnowGuard: Knowledge-Driven Abstention for Multi-Round Clinical Reasoning](https://mlsanity.com/paper/2509.24816/) (2025)
- [Know What You do Not Know: Verbalized Uncertainty Estimation Robustness on Corrupted Images in Vision-Language Models](https://mlsanity.com/paper/2504.03440/) (2025)
- [Variational Visual Question Answering for Uncertainty-Aware Selective Prediction](https://mlsanity.com/paper/2505.09591/) (2025)
- [The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Models](https://mlsanity.com/paper/2603.16728/) (2026)
- [Transparent and Coherent Procedural Mistake Detection](https://mlsanity.com/paper/2412.11927/) (2024)
- [SafeCoT: Improving VLM Safety with Minimal Reasoning](https://mlsanity.com/paper/2506.08399/) (2025)
- [Can VLMs Recall Factual Associations From Visual References?](https://mlsanity.com/paper/2508.18297/) (2025)
- [HARMONY: Hidden Activation Representations and Model Output-Aware Uncertainty Estimation for Vision-Language Models](https://mlsanity.com/paper/2510.22171/) (2025)
- [VLM-UQBench: A Benchmark for Modality-Specific and Cross-Modality Uncertainties in Vision Language Models](https://mlsanity.com/paper/2602.09214/) (2026)
- [Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Model](https://mlsanity.com/paper/2606.16667/) (2026)
- [Knowing When to Answer: Adaptive Confidence Refinement for Reliable Audio-Visual Question Answering](https://mlsanity.com/paper/2602.04924/) (2026)
- [Leveraging Data to Say No: Memory Augmented Plug-and-Play Selective Prediction](https://mlsanity.com/paper/2601.22570/) (2026)
- [Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks](https://mlsanity.com/paper/2602.06526/) (2026)
- [Geometric Risk Control for Vision-Language Model OCR](https://mlsanity.com/paper/2603.19790/) (2026)
- [SIEVES: Selective Prediction Generalizes through Visual Evidence Scoring](https://mlsanity.com/paper/2604.25855/) (2026)
- [AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficulty](https://mlsanity.com/paper/2608.16690/) (2026)
- [Improving VQA Reliability: A Dual-Assessment Approach with Self-Reflection and Cross-Model Verification](https://mlsanity.com/paper/2512.14770/) (2025)
- [Learning When Not to Decide: A Framework for Overcoming Factual Presumptuousness in AI Adjudication](https://mlsanity.com/paper/2604.19895/) (2026)
- [A Regime Theory of Controller Class Selection for LLM Action Decisions](https://mlsanity.com/paper/2605.06339/) (2026)
- [Uncertainty Quantification for Multimodal Retrieval Augmented Generation](https://mlsanity.com/paper/2605.29956/) (2026)
- [It Takes One to Bias Them All: Breaking Bad with One-Shot GRPO](https://mlsanity.com/paper/2606.10931/) (2026)
- [When Depth Is Better Told Than Shown: Depth-Ordinal Prompting for Vision-Language Spatial Reasoning](https://mlsanity.com/paper/2607.11173/) (2026)
- [When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering](https://mlsanity.com/paper/2608.05670/) (2026)
- [Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading](https://mlsanity.com/paper/2608.05675/) (2026)
- [How Do VLMs Behave When Blind or Misled? Behavioral Evaluation of VLMs on Scientific Figures](https://mlsanity.com/paper/2608.13267/) (2026)

## Abstract (arXiv)

Selective prediction minimizes incorrect predictions from vision-language models (VLMs) by allowing them to abstain from answering when uncertain. However, when deploying a vision-language system with low tolerance for inaccurate predictions, selective prediction may be over-cautious and abstain too frequently, even on many correct predictions. We introduce ReCoVERR, an inference-time algorithm to reduce the over-abstention of a selective vision-language system without increasing the error rate of the system's predictions. When the VLM makes a low-confidence prediction, instead of abstaining ReCoVERR tries to find relevant clues in the image that provide additional evidence for the prediction. ReCoVERR uses an LLM to pose related questions to the VLM, collects high-confidence evidences, and if enough evidence confirms the prediction the system makes a prediction instead of abstaining. ReCoVERR enables three VLMs (BLIP2, InstructBLIP, and LLaVA-1.5) to answer up to 20% more questions on the VQAv2 and A-OKVQA tasks without decreasing system accuracy, thus improving overall system reliability. Our code is available at https://github.com/tejas1995/ReCoVERR.