# MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements

arXiv: https://arxiv.org/abs/2511.13087 · HTML: https://mlsanity.com/paper/2511.13087/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 17 Nov 2025.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 19 of 19)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024)
- [ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use](https://mlsanity.com/paper/2504.07981/) (2025)
- [Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs](https://mlsanity.com/paper/2404.05719/) (2024)
- [Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis](https://mlsanity.com/paper/2505.13227/) (2025)
- [GUICourse: From General Vision Language Models to Versatile GUI Agents](https://mlsanity.com/paper/2406.11317/) (2024)
- [GTA1: GUI Test-time Scaling Agent](https://mlsanity.com/paper/2507.05791/) (2025)
- [GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents](https://mlsanity.com/paper/2506.03143/) (2025)
- [Screen2Vec: Semantic Embedding of GUI Screens and GUI Components](https://mlsanity.com/paper/2101.11103/) (2021)
- [DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning](https://mlsanity.com/paper/2507.00008/) (2025)
- [Visual Test-time Scaling for GUI Agent Grounding](https://mlsanity.com/paper/2505.00684/) (2025)
- [Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systems](https://mlsanity.com/paper/2503.06470/) (2025)
- [UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis](https://mlsanity.com/paper/2504.11257/) (2025)
- [GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration](https://mlsanity.com/paper/2501.13896/) (2025)
- [ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search](https://mlsanity.com/paper/2505.15259/) (2025)

## Built on by (most cited first, 3 of 3)

- [Zoom Consistency: A Free Confidence Signal in Multi-Step Visual Grounding Pipelines](https://mlsanity.com/paper/2604.15376/) (2026)
- [Software Engineering for and with GUI Agent](https://mlsanity.com/paper/2608.09278/) (2026)
- [MHVF: Training-Free GUI Grounding via Verification-Guided Multi-Hypothesis Search](https://mlsanity.com/paper/2610.17792/) (2026)

## Abstract (arXiv)

Graphical User Interface (GUI) grounding - the task of mapping natural language instructions to screen coordinates - is essential for autonomous agents and accessibility technologies. Existing systems rely on monolithic models or one-shot pipelines that lack modularity and fail under visual clutter and ambiguous instructions. We introduce MEGA-GUI, a multi-stage framework that separates grounding into coarse Region-of-Interest (ROI) selection and fine-grained element grounding, orchestrated by specialized vision-language agents. MEGA-GUI features a bidirectional ROI zoom algorithm that mitigates spatial dilution and a context-aware rewriting agent that reduces semantic ambiguity. Our analysis reveals complementary strengths and weaknesses across vision-language models at different visual scales, and we show that leveraging this modular structure achieves consistently higher accuracy than monolithic approaches. On the visually dense ScreenSpot-Pro benchmark, MEGA-GUI attains 73.18% accuracy, and on the semantically complex OSWorld-G benchmark it reaches 68.63%, surpassing previously reported results. Code and the Grounding Benchmark Toolkit (GBT) are available at https://github.com/samsungsds-research-papers/mega-gui.