mlsanityData to 7 Oct 2026

Decision-Aware Uncertainty Evaluation of Vision-Language Model-Based Early Action Anticipation for Human-Robot Interaction

arXiv 2603.10061 ↗First posted 9 Mar 2026cs.ROcited by 2

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Robots in shared workspaces must interpret human actions from partial, ambiguous observations, where overconfident early predictions can lead to unsafe or disruptive interaction. This challenge is amplified in egocentric views, where viewpoint changes and occlusions increase perceptual noise and ambiguity. As a result, downstream human-robot interaction modules require not only an action hypothesis but also a trustworthy estimate of confidence under partial observation. Recent vision-language model-based approaches have been proposed for short-term action recognition due to their open-vocabulary and context-aware reasoning, but their uncertainty reliability in the temporal-prefix regime is largely uncharacterized. We present the first systematic evaluation of uncertainty in vision-language model-based short-term action recognition for human-robot interaction. We introduce a temporal-prefix evaluation protocol and metrics for calibration and selective prediction. We also characterize miscalibration patterns and failure modes under partial observations. Our study provides the missing reliability evidence needed to use vision-language model predictions in confidence-gated human-robot interaction modules.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 6 of its 33 references that are in the map. First posted on arXiv 9 Mar 2026.

Builds on

6 of 33 references.

References · 6

  1. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  3. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  4. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  5. 2024A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimatorscited by 13
  6. 2025Enhancing accuracy of uncertainty estimation in appearance-based gaze tracking with probabilistic evaluation and calibrationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Task-Driven 3D Printability Assistance via Geometry- and Knowledge-Grounded LLM Reasoningcited by 0
  2. 2026Language-Augmented Video Action Anticipation: Design Fundamentals, Benchmarks, and Open Challengescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Perturbation-Based Epistemic Uncertainty for Failure Detection in Vision-Language-Action Models
  2. 2026Uncertainty Quantification for Flow-Based Vision-Language-Action Models
  3. 2025Confidence Calibration in Vision-Language-Action Models
  4. 2026LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries
  5. 2026LM-X: Explainable Vision--Language--Action Modeling via Progress, Event, and Uncertainty Prediction
  6. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
  7. 2026FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty
  8. 2026ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Control