Visual Grounding Methods for Efficient Interaction with Desktop Graphical User Interfaces
Not on the map. Why?
Abstract (arXiv)
Most visual grounding solutions primarily focus on realistic images. However, applications involving synthetic images, such as Graphical User Interfaces (GUIs), remain limited. This restricts the development of autonomous computer vision-powered artificial intelligence (AI) agents for automatic application interaction. Enabling AI to effectively understand and interact with GUIs is crucial to advancing automation in software testing, accessibility, and human-computer interaction. In this work, we explore Instruction Visual Grounding (IVG), a multi-modal approach to object identification within a GUI. More precisely, given a natural language instruction and a GUI screen, IVG locates the coordinates of the element on the screen where the instruction should be executed. We propose two main methods: (1) IVGocr, which combines a Large Language Model (LLM), an object detection model, and an Optical Character Recognition (OCR) module; and (2) IVGdirect, which uses a multimodal architecture for end-to-end grounding. For each method, we introduce a dedicated dataset. In addition, we propose the Central Point Validation (CPV) metric, a relaxed variant of the classical Central Proximity Score (CPS) metric. Our final test dataset is publicly released to support future research.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 72 of its 88 references that are in the map. First posted on arXiv 5 May 2024.
Builds on
72 of 88 references.
References · 72
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2023Visual Instruction Tuningcited by 7,792
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2020End-to-End Object Detection with Transformerscited by 7,384
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
38 more
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2021Zero-Shot Text-to-Image Generationcited by 4,276
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
- 2022A ConvNet for the 2020scited by 3,545
- 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
- 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
- 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
- 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
- 2016Modeling Context in Referring Expressionscited by 1,054
- 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
- 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
- 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
- 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 812
- 2021Florence: A New Foundation Model for Computer Visioncited by 802
- 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
- 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
- 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
- 2023Segment Everything Everywhere All at Oncecited by 546
- 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
- 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
- 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
- 2021Conditional DETR for Fast Training Convergencecited by 422
- 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
- 2023Language Models can Solve Computer Taskscited by 371
- 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
- 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
- 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
- 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
50 most cited of 72 All 72 (JSON)
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.