mlsanityData to 14 Oct 2026

How Many Pixels Is a Digit Worth? Place-Aware Coordinate Entropy for GUI Agent Confidence Estimation

arXiv 2609.24277 ↗First posted 21 Sep 2026cs.AIcited by 0

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

GUI agents predict click coordinates as digit-token sequences, but standard text-LLM confidence estimation methods rank correct clicks from wrong ones only weakly. GUI-specific alternatives use K samples or new supervision, but still leave room for improvement. We trace part of this to place-value asymmetry: bounding-box correctness often makes higher-place digits more important than lower-place digits, so uniform aggregation weakens the signal that determines correctness. The fix is to weight each digit's Shannon entropy by its place value. We call this Place-Aware Coordinate Entropy (PACE). Across fixed-scale agents on ScreenSpot-Pro and ScreenSpot-v2, PACE wins both AUROC and selective accuracy on all primary comparisons in a single forward pass, matching or outperforming K-sample baselines at a fraction of the cost. PACE provides a per-click confidence estimate that turns coordinate-token internals into a practical confidence signal for GUI agent deployment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 19 of its 26 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

19 of 26 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec
  2. 2025Understanding GUI Agent Localization Biases through Logit Sharpness18 Jun
  3. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
  4. 2025Towards Trustworthy GUI Agents: A Survey30 Mar
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  6. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  7. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 12

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  3. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  4. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  5. 2017Selective Classification for Deep Neural Networkscited by 852
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  7. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  8. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  9. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  10. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  11. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 53
  12. 2026SafeGround: Know When to Trust GUI Grounding Models via Uncertainty Calibrationcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding
  2. 2026Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
  3. 2025Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning
  4. 2025Test-Time Reinforcement Learning for GUI Grounding via Region Consistency
  5. 2026GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding
  6. 2025LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization
  7. 2026Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding
  8. 2025Improving GUI Grounding with Explicit Position-to-Coordinate Mapping