Co-EPG: A Framework for Co-Evolution of Planning and Grounding in Autonomous GUI Agents
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
Graphical User Interface (GUI) task automation constitutes a critical frontier in artificial intelligence research. While effective GUI agents synergistically integrate planning and grounding capabilities, current methodologies exhibit two fundamental limitations: (1) insufficient exploitation of cross-model synergies, and (2) over-reliance on synthetic data generation without sufficient utilization. To address these challenges, we propose Co-EPG, a self-iterative training framework for Co-Evolution of Planning and Grounding. Co-EPG establishes an iterative positive feedback loop: through this loop, the planning model explores superior strategies under grounding-based reward guidance via Group Relative Policy Optimization (GRPO), generating diverse data to optimize the grounding model. Concurrently, the optimized Grounding model provides more effective rewards for subsequent GRPO training of the planning model, fostering continuous improvement. Co-EPG thus enables iterative enhancement of agent capabilities through self-play optimization and training data distillation. On the Multimodal-Mind2Web and AndroidControl benchmarks, our framework outperforms existing state-of-the-art methods after just three iterations without requiring external data. The agent consistently improves with each iteration, demonstrating robust self-enhancement capabilities. This work establishes a novel training paradigm for GUI agents, shifting from isolated optimization to an integrated, self-driven co-evolution approach.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 36 of its 49 references that are in the map. First posted on arXiv 13 Nov 2025.
Builds on
36 of 49 references.
Closest ancestors · 11
Same direction, newest first
- 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning18 May
- 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
- 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents1 Apr
- 2025COLA: A Scalable Multi-Agent Framework For Windows UI Task Automation12 Mar
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection8 Jan
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
- 2024ScreenAgent: A Vision Language Model-driven Computer Control Agent9 Feb
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
Other references · 25
- 2023GPT-4 Technical Reportcited by 18,132
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2023Qwen Technical Reportcited by 3,273
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
- 2024DeepSeek-V3 Technical Reportcited by 724
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
- 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
- 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
- 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
- 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
13 more
- 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
- 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
- 2024OmniParser for Pure Vision Based GUI Agentcited by 193
- 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
- 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
- 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 123
- 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
- 2025WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Modelcited by 56
- 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
- 2025Learning to Contextualize Web Pages for Enhanced Decision Making by LLM Agentscited by 16
- 2025LiteWebAgent: The Open-Source Suite for VLM-Based Web-Agent Applicationscited by 14
- 2025WinClick: GUI Grounding with Multimodal Large Language Modelscited by 11
- 2025Importance-Aware Data Selection for Efficient LLM Instruction Tuningcited by 7
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution
- 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
- 2025CRAFT-GUI: Curriculum-Reinforced Agent For GUI Tasks
- 2024AutoGLM: Autonomous Foundation Agents for GUIs
- 2025UItron: Foundational GUI Agent with Advanced Perception and Planning
- 2026Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation
- 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
- 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience