POINTS-GUI-G: GUI-Grounding Journey
Not on the map. Why?
Abstract (arXiv)
The rapid advancement of vision-language models has catalyzed the emergence of GUI agents, which hold immense potential for automating complex tasks, from online shopping to flight booking, thereby alleviating the burden of repetitive digital workflows. As a foundational capability, GUI grounding is typically established as a prerequisite for end-to-end task execution. It enables models to precisely locate interface elements, such as text and icons, to perform accurate operations like clicking and typing. Unlike prior works that fine-tune models already possessing strong spatial awareness (e.g., Qwen3-VL), we aim to master the full technical pipeline by starting from a base model with minimal grounding ability, such as POINTS-1.5. We introduce POINTS-GUI-G-8B, which achieves state-of-the-art performance with scores of 59.9 on ScreenSpot-Pro, 66.0 on OSWorld-G, 95.7 on ScreenSpot-v2, and 49.9 on UI-Vision. Our model's success is driven by three key factors: (1) Refined Data Engineering, involving the unification of diverse open-source datasets format alongside sophisticated strategies for augmentation, filtering, and difficulty grading; (2) Improved Training Strategies, including continuous fine-tuning of the vision encoder to enhance perceptual accuracy and maintaining resolution consistency between training and inference; and (3) Reinforcement Learning (RL) with Verifiable Rewards. While RL is traditionally used to bolster reasoning, we demonstrate that it significantly improves precision in the perception-intensive GUI grounding task. Furthermore, GUI grounding provides a natural advantage for RL, as rewards are easily verifiable and highly accurate.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 62 of its 69 references that are in the map. First posted on arXiv 6 Feb 2026.
Builds on
62 of 69 references.
References · 62
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2023Visual Instruction Tuningcited by 7,792
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
38 more
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2023DataComp: In search of the next generation of multimodal datasetscited by 635
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
- 2025Kimi-VL Technical Reportcited by 381
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
- 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 294
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
- 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 240
- 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
- 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
- 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
- 2024OmniParser for Pure Vision Based GUI Agentcited by 193
- 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
- 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
- 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
- 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
- 2025GTA1: GUI Test-time Scaling Agentcited by 102
- 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
- 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
- 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
- 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 79
- 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
- 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
- 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 65
- 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
- 2023WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semanticscited by 45
- 2025Step-GUI Technical Reportcited by 38
50 most cited of 62 All 62 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026DashAct: A Progressive Diagnostic Benchmark for GUI Agents in Interactive Dashboard Analysis26 Sep 2026
- 2026Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding14 Sep 2026
- 2026One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding29 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2025MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements
- 2025Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding
- 2026AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement
- 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedback
- 2024Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding
- 2026GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
- 2026GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction
- 2025Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Grounding