Zoom Consistency: A Free Confidence Signal in Multi-Step Visual Grounding Pipelines
Not on the map. Why?
Abstract (arXiv)
Multi-step zoom-in pipelines are widely used for GUI grounding, yet the intermediate predictions they produce are typically discarded after coordinate remapping. We observe that these intermediate outputs contain a useful confidence signal for free: zoom consistency, the distance between a model's step-2 prediction and the crop center. Unlike log-probabilities or token-level uncertainty, zoom consistency is a geometric quantity in a shared coordinate space, making it directly comparable across architecturally different VLMs without calibration. We prove this quantity is a linear estimator of step-1 spatial error under idealized conditions (perfect step-2, target within crop) and show it correlates with prediction correctness across two VLMs (AUC = 0.60; Spearman rho = -0.14, p < 10^{-6} for KV-Ground-8B; rho = -0.11, p = 0.0003 for Qwen3.5-27B). The correlation is small but consistent across models, application categories, and operating systems. As a proof-of-concept, we use zoom consistency to route between a specialist and generalist model, capturing 16.5% of the oracle headroom between them (+0.8%, McNemar p = 0.19). Code is available at https://github.com/omxyz/zoom-consistency-routing.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 16 of its 19 references that are in the map. First posted on arXiv 15 Apr 2026.
Builds on
16 of 19 references.
References · 16
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 254
- 2025Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMscited by 96
- 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95
- 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 81
- 2025Visual Test-time Scaling for GUI Agent Groundingcited by 30
- 2025Test-Time Reinforcement Learning for GUI Grounding via Region Consistencycited by 21
- 2025Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Groundingcited by 8
4 more
- 2026Adaptive Vision-Language Model Routing for Computer Use Agentscited by 8
- 2026AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinementcited by 6
- 2025Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMscited by 4
- 2025MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elementscited by 3
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026TACO: Tool-Augmented Credit Optimization for Agentic Tool Usecited by 5
- 2026It's the Decoding Format, Not the Perturbation: Auditing Consistency-Based Selection for Vision-Language Test-Time Scalingcited by 0
- 2026When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Renderingcited by 0
- 2026Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Readingcited by 0
- 2026Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading6 Aug 2026
- 2026When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering6 Aug 2026
- 2026It's the Decoding Format, Not the Perturbation: Auditing Consistency-Based Selection for Vision-Language Test-Time Scaling2 Aug 2026
- 2026TACO: Tool-Augmented Credit Optimization for Agentic Tool Use29 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models
- 2026LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection
- 2026One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding
- 2026AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding
- 2026GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
- 2026AgroVG: A Large-Scale Multi-Source Benchmark for Agricultural Visual Grounding
- 2025Improving GUI Grounding with Explicit Position-to-Coordinate Mapping
- 2026Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets