# Unified Multimodal Uncertain Inference

arXiv: https://arxiv.org/abs/2604.08701 · HTML: https://mlsanity.com/paper/2604.08701/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 58 references that are in the map.
- First posted on arXiv 9 Apr 2026.

## Builds on (most cited first, 24 of 24)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [Visual Entailment: A Novel Task for Fine-Grained Image Understanding](https://mlsanity.com/paper/1901.06706/) (2019)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty](https://mlsanity.com/paper/2507.16806/) (2025)
- [Improving Regression Performance with Distributional Losses](https://mlsanity.com/paper/1806.04613/) (2018)
- [VIOLIN: A Large-Scale Dataset for Video-and-Language Inference](https://mlsanity.com/paper/2003.11618/) (2020)
- [Audio Entailment: Assessing Deductive Reasoning for Audio Understanding](https://mlsanity.com/paper/2407.18062/) (2024)
- [MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval](https://mlsanity.com/paper/2410.11619/) (2024)
- [Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answering](https://mlsanity.com/paper/2502.13962/) (2025)
- [Zero and Few-shot Semantic Parsing with Ambiguous Inputs](https://mlsanity.com/paper/2306.00824/) (2023)
- [WikiVideo: Article Generation from Multiple Videos](https://mlsanity.com/paper/2504.00939/) (2025)
- [Ambiguous Images With Human Judgments for Robust Visual Event Classification](https://mlsanity.com/paper/2210.03102/) (2022)
- [Investigating the Histogram Loss in Regression](https://mlsanity.com/paper/2402.13425/) (2024)
- [Conformal Thinking: Risk Control for Reasoning on a Compute Budget](https://mlsanity.com/paper/2602.03814/) (2026)
- [Always Tell Me The Odds: Fine-grained Conditional Probability Estimation](https://mlsanity.com/paper/2505.01595/) (2025)
- [Garden-Path Traversal in GPT-2](https://mlsanity.com/paper/2205.12302/) (2022)

## Built on by (most cited first, 3 of 3)

- [TRACE: Evidence Grounding-Guided Multi-Video Event Understanding and Claim Generation](https://mlsanity.com/paper/2605.16740/) (2026)
- [CRAFT: Critic-Refined Adaptive Key-Frame Targeting for Multimodal Video Question Answering](https://mlsanity.com/paper/2605.19075/) (2026)
- [Decoupling Semantics and Logic: A Training-Free Coarse-to-Fine Pipeline for Video Retrieval-Augmented Generation](https://mlsanity.com/paper/2606.07924/) (2026)

## Abstract (arXiv)

We introduce Unified Multimodal Uncertain Inference (UMUI), a multimodal inference task spanning text, audio, and video, where models must produce calibrated probability estimates of hypotheses conditioned on a premise in any modality or combination. While uncertain inference has been explored in text, extension to other modalities has been limited to single-modality binary entailment judgments, leaving no framework for fine-grained probabilistic reasoning in or across other modalities. To address this, we curate a human-annotated evaluation set with scalar probability judgments across audio, visual, and audiovisual settings, and additionally evaluate on existing text and audio benchmarks. We introduce CLUE (Calibrated Latent Uncertainty Estimation), which combines self-consistent teacher calibration and distribution-based confidence probing to produce calibrated predictions. We demonstrate that our 3B-parameter model achieves equivalent or stronger performance than zero-shot baselines up to 32B parameters across all modalities.