mlsanityData to 14 Oct 2026

Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets

arXiv 2606.25760 ↗First posted 24 Jun 2026cs.LG cs.AI cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

also in 1 more
Abstract (arXiv)

Computer-use agents turn vision-language model (VLM) predictions into executable GUI clicks, so reliable uncertainty estimates are essential for rejection, calibration, miss-severity ranking, and spatial safety regions. Yet evidence on post-hoc uncertainty quantification (UQ) for these agents is fragmented across isolated model and dataset pairs, leaving it unclear whether UQ rankings stay stable when the agent, benchmark, or observable interface changes. We present Argus, a cross-regime benchmark for post-hoc UQ in single-step executable GUI grounding: a 27-method open-weight matrix over 4 VLM agents and 4 datasets, plus an 8-method closed-source matrix across 3 frontier vendors where logits, hidden states, and attention maps are unavailable. Evaluated methods span logit-based scores, sampling and consistency measures, hidden-state and density estimators (Mahalanobis, SAPLMA), attention-based scores, P(True) and verbalised-confidence prompting, and split-conformal prediction. The main finding is selective transfer: UQ rankings are stable across datasets for a fixed model, but degrade across model classes and observable interfaces. Hidden-state and density methods are the most stable open-weight family, while CoCoA-1MCA, Focus, sampling-based scores, and verbalised self-assessment win in specific regimes. Within-model ranking transfer is strong (Spearman rho up to 0.969), but cross-tier transfer to closed-source vendors averages only +0.08, so closed-source UQ should be reranked on the target rather than extrapolated. Conformal click regions show score-level discrimination is not enough for deployment: locally weighted disks shrink radii by 40-60% when the plug-in UQ is calibrated, but coverage degrades under calibration-test or interface mismatch. We release per-item records, calibration/test splits, UQ scores, and analysis scripts for regime-aware UQ selection in GUI agents.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 58 references that are in the map. First posted on arXiv 24 Jun 2026.

Builds on

51 of 58 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025The Art of Saying "Maybe": A Conformal Lens for Uncertainty Benchmarking in VLMs16 Sep
  2. 2025Learning Conformal Abstention Policies for Adaptive Risk Management in Large Language and Vision-Language Models8 Feb
  3. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimation18 Nov
  4. 2024Uncertainty-Aware Evaluation for Vision-Language Models22 Feb

Other references · 47

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  5. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  6. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  7. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  8. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  9. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  10. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  11. 2016Distribution-Free Predictive Inference For Regressioncited by 668
  12. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
35 more
  1. 2019Conformalized Quantile Regressioncited by 629
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  3. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  4. 2018Predictive Uncertainty Estimation via Prior Networkscited by 557
  5. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  6. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
  7. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
  8. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  9. 2022Conformal Risk Controlcited by 299
  10. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
  11. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 259
  12. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 254
  13. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 197
  14. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
  15. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 139
  16. 2018Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flowcited by 135
  17. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 131
  18. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  19. 2023Enhancing Uncertainty-Based Hallucination Detection with Stronger Focuscited by 99
  20. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 77
  21. 2025Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistencycited by 30
  22. 2024Adaptive Bounding Box Uncertainties via Two-Step Conformal Predictioncited by 23
  23. 2024WebSuite: Systematically Evaluating Why Web Agents Failcited by 16
  24. 2025Language Model Uncertainty Quantification with Attention Chaincited by 14
  25. 2026TRACER: Trajectory Risk Aggregation for Critical Episodes in Agentic Reasoningcited by 11
  26. 2026UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Groundingcited by 8
  27. 2026VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluationcited by 8
  28. 2025Learnable Conformal Prediction with Context-Aware Nonconformity Functions for Robotic Planning and Perceptioncited by 7
  29. 2025Beyond Confidence: Adaptive Abstention in Dual-Threshold Conformal Prediction for Autonomous System Perceptioncited by 6
  30. 2025V2P: Visual Attention Calibration for GUI Grounding via Background Suppression and Center Peakingcited by 6
  31. 2025Calibrated Decomposition of Aleatoric and Epistemic Uncertainty in Deep Features for Inference-Time Adaptationcited by 6
  32. 2026SafeGround: Know When to Trust GUI Grounding Models via Uncertainty Calibrationcited by 6
  33. 2025Torch-Uncertainty: A Deep Learning Framework for Uncertainty Quantificationcited by 4
  34. 2026Uncertainty-Guided Inference-Time Depth Adaptation for Transformer-Based Visual Trackingcited by 3
  35. 2026TRIAGE: Type-Routed Interventions via Aleatoric-Epistemic Gated Estimation in Robotic Manipulation and Adaptive Perception -- Don't Treat All Uncertainty the Samecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models
  2. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation
  3. 2026VLM-UQBench: A Benchmark for Modality-Specific and Cross-Modality Uncertainties in Vision Language Models
  4. 2026FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty
  5. 2025R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding
  6. 2026VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluation
  7. 2026Dynamic Distribution-Aware Uncertainty Tracking in Vision-Language Representation Learning
  8. 2026Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption