Towards Scalable Lightweight GUI Agents via Multi-role Orchestration
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 1 more
Abstract (arXiv)
Autonomous Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) enable digital automation on end-user devices. While scaling both parameters and data has yielded substantial gains, advanced methods still suffer from prohibitive deployment costs on resource-constrained devices. When facing complex in-the-wild scenarios, lightweight GUI agents are bottlenecked by limited capacity and poor task scalability under end-to-end episodic learning, impeding adaptation to multi-agent systems (MAS), while training multiple skill-specific experts remains costly. Can we strike an effective trade-off in this cost-scalability dilemma, enabling lightweight MLLMs to participate in realistic GUI workflows? To address these challenges, we propose the LAMO framework, which endows a lightweight MLLM with GUI-specific knowledge and task scalability, allowing multi-role orchestration to expand its capability boundary for GUI automation. LAMO combines role-oriented data synthesis with a two-stage training recipe: (i) supervised fine-tuning with Perplexity-Weighted Cross-Entropy optimization for knowledge distillation and visual perception enhancement, and (ii) reinforcement learning for role-oriented cooperative exploration. With LAMO, we develop a task-scalable native GUI agent, LAMO-3B, supporting monolithic execution and MAS-style orchestration. When paired with advanced planners as a plug-and-play policy executor, LAMO-3B can continuously benefit from planner advances, enabling a higher performance ceiling. Extensive static and online evaluations validate the effectiveness of our design.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 52 references that are in the map. First posted on arXiv 15 Apr 2026.
Builds on
31 of 52 references.
Closest ancestors · 6
Same direction, newest first
- 2025History-Aware Reasoning for GUI Agents12 Nov
- 2025PG-Agent: An Agent Powered by Page Graph27 Aug
- 2025R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding8 Jul
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
Other references · 25
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
- 2025Memory in the Age of AI Agentscited by 285
- 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
- 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
- 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
- 2024On the Effects of Data Scale on UI Control Agentscited by 187
13 more
- 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
- 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
- 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
- 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
- 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
- 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 80
- 2025A Survey on (M)LLM-Based GUI Agentscited by 61
- 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 20
- 2025UGround: Towards Unified Visual Grounding with Unrolled Transformerscited by 14
- 2025ProBench: Benchmarking GUI Agents with Accurate Process Informationcited by 14
- 2025MP-GUI: Modality Perception with MLLMs for GUI Understandingcited by 12
- 2025Co-EPG: A Framework for Co-Evolution of Planning and Grounding in Autonomous GUI Agentscited by 3
- 2025Towards Scalable Web Accessibility Audit with MLLMs as Copilotscited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learning
- 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
- 2026LiteGUI: Distilling Compact GUI Agents with Reinforcement Learning
- 2025UIPro: Unleashing Superior Interaction Capability For GUI Agents
- 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interaction
- 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
- 2026Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization
- 2024TinyClick: Single-Turn Agent for Empowering GUI Automation