mlsanityData to 7 Oct 2026

Witness Evidence Portfolios: Single-Prefill Risk Detection for Closed Multimodal Answers

arXiv 2607.27667 ↗First posted 30 Jul 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Reliable deployment of multimodal large language models (MLLMs) requires deciding whether a confident visual answer should be trusted, reviewed, or routed to a stronger system. Confidence scores capture candidate margins, but not where the estimated signed visual readouts associated with those margins come from or how they are distributed. We study inference-time risk detection for closed visual answers using the same white-box prefill path that produces the answer. Witness Evidence Portfolios (WEP) first estimates, layer by layer, which visual contributions support or contradict the predicted candidate. It summarizes these contributions through two interpretable route families: question-related evidence provenance and signed evidence concentration. Nested grouped validation chooses the more reliable family and a sparse top-k route portfolio, which is fused with candidate confidence. WEP needs no image perturbation, decoding change, backward pass, or external verifier. Across three MLLMs and four binary-answer benchmarks, WEP improves mean error AP by 0.134. All 12 model--dataset gains are positive, and image-cluster bootstrap intervals are strictly positive on 10 pairs. WEP targets white-box closed-answer systems and uses a labeled calibration slice.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 40 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

27 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Variational Visual Question Answering for Uncertainty-Aware Selective Prediction14 May

Other references · 26

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  8. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  9. 2019Attention is not Explanationcited by 916
  10. 2017Selective Classification for Deep Neural Networkscited by 799
  11. 2018Object Hallucination in Image Captioningcited by 661
  12. 2019Attention is not not Explanationcited by 629
14 more
  1. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  2. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  3. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  4. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  5. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 198
  6. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 86
  7. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  8. 2023Improving Selective Visual Question Answering by Learning from Your Peerscited by 24
  9. 2026HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Tokencited by 13
  10. 2026VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneckcited by 8
  11. 2025HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Modelscited by 5
  12. 2025Taming Object Hallucinations with Verified Atomic Confidence Estimationcited by 4
  13. 2025LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systemscited by 4
  14. 2025Reallocating Attention Across Layers to Reduce Multimodal Hallucinationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios
  2. 2026CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation
  3. 2026Layers, Sinks, and Scaling: Adaptive Evidence Selection for Multimodal Large Language Models
  4. 2026Q-CueGraph: Query-Conditioned Visual Evidence Graphs for Multimodal Reasoning
  5. 2026VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence
  6. 2026Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning
  7. 2026Latent Noise Mask for Reducing Visual Redundancy in Multimodal Large Language Models
  8. 2026Composition, Not Conversation: VLMs Lose the Scene, Not the Thread