mlsanityData to 14 Oct 2026

HalluPrism: When Multimodal Uncertainty Should Diagnose, Not Decide

arXiv 2608.29193 ↗First posted 29 Aug 2026cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) can assign similar confidence to answers that fail for different reasons. We propose HalluPrism, a behavioral diagnostic that re-runs an answer after visual degradation, blank-image replacement, and grounding or relation checks. These targeted probes yield a signature over visual-perturbation sensitivity (V ), image-removal confidence retention (L), and grounding/relation-probe instability (A). Across 58K+ examples from four benchmarks and four MLLMs, image-removal confidence retention is most prevalent, while grounding/relation-probe instability better separates failure families. Only 18 of 48 source-target checks are diagonally aligned, so the coordinates should be interpreted jointly rather than as independent causal sources. With the dataset fixed, the joint signature improves failure-family AUROC from 0.634 to 0.769 on HallusionBench and from 0.707 to 0.817 on VizWiz, with smaller gains on POPE and VSR. In pooled XGBoost analysis, AUROC rises from 0.78 with scalar confidence to 0.95 with (V, L, A) and 0.97 when confidence is added. The same signature does not automatically improve correctness ranking. The three tested direct scalarizations can harm it. These results separate failure diagnosis from abstention scoring: multimodal uncertainty should characterize failure structure before it is used to decide whether to abstain or correct.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 42 references that are in the map. First posted on arXiv 29 Aug 2026.

Builds on

11 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Qwen3-VL Technical Report26 Nov
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 8

  1. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  2. 2017Selective Classification for Deep Neural Networkscited by 852
  3. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  5. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  6. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigationcited by 23
  7. 2025Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibrationcited by 11
  8. 2025On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Modelscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Mitigating Visual Hallucinations in Multimodal Systems through Retrieval-Augmented Reliability-Aware Inference
  2. 2026SAVOR: Self-Aware Visual Grounding via Confidence-Calibrated Reinforcement Learning for Multimodal Hallucination Mitigation
  3. 2025Taming Object Hallucinations with Verified Atomic Confidence Estimation
  4. 2026Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation
  5. 2025SegSub: Evaluating Robustness to Knowledge Conflicts and Hallucinations in Vision-Language Models
  6. 2025Measuring Epistemic Humility in Multimodal Large Language Models
  7. 2025Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability
  8. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models