Lexical Coupling in GUI Element Grounding: Sentence Embeddings Track Labels across Mobile and Web
Not on the map. Why?
Abstract (arXiv)
GUI grounding evaluations that expose UI elements as text metadata often treat high instruction-element embedding similarity as evidence of semantic grounding. Across three mobile and web benchmarks, we show that this interpretation is frequently confounded by visible-label recovery. Lexical baselines remain competitive at top-1, label-poor targets remain weak for text-only methods, and encoder top-1 hits are predictable from lexical rank, candidate-pool size, and label type. We evaluate each action as a same-screen ranking task, comparing five off-the-shelf single-vector encoders with lexical baselines. Encoders recover some lexical misses, but deployable fusion gains are much smaller than target-aware oracle gains. These findings show that embedding-based evaluations can conflate visible-label recovery with semantic GUI grounding. Embedding-based evaluations should therefore report lexical baselines, label-type stratification, and deployable-fusion diagnostics. Our released repository provides analysis scripts and detexted per-step panels: https://github.com/qijia123/lexical-coupling-release.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 30 references that are in the map. First posted on arXiv 22 Aug 2026.
Builds on
16 of 30 references.
References · 16
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
- 2020Shortcut Learning in Deep Neural Networkscited by 1,969
- 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
- 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
- 2018Annotation Artifacts in Natural Language Inference Datacited by 671
- 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
- 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 286
4 more
- 2024On the Effects of Data Scale on UI Control Agentscited by 194
- 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agentscited by 107
- 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
- 2025AndroidControl-Curated: Revealing the True Potential of GUI Agents through Benchmark Purificationcited by 6
Built on by
Not cited yet.