mlsanityData to 7 Oct 2026

Co-EPG: A Framework for Co-Evolution of Planning and Grounding in Autonomous GUI Agents

arXiv 2511.10705 ↗First posted 13 Nov 2025cs.AI cs.CLcited by 3

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Graphical User Interface (GUI) task automation constitutes a critical frontier in artificial intelligence research. While effective GUI agents synergistically integrate planning and grounding capabilities, current methodologies exhibit two fundamental limitations: (1) insufficient exploitation of cross-model synergies, and (2) over-reliance on synthetic data generation without sufficient utilization. To address these challenges, we propose Co-EPG, a self-iterative training framework for Co-Evolution of Planning and Grounding. Co-EPG establishes an iterative positive feedback loop: through this loop, the planning model explores superior strategies under grounding-based reward guidance via Group Relative Policy Optimization (GRPO), generating diverse data to optimize the grounding model. Concurrently, the optimized Grounding model provides more effective rewards for subsequent GRPO training of the planning model, fostering continuous improvement. Co-EPG thus enables iterative enhancement of agent capabilities through self-play optimization and training data distillation. On the Multimodal-Mind2Web and AndroidControl benchmarks, our framework outperforms existing state-of-the-art methods after just three iterations without requiring external data. The agent consistently improves with each iteration, demonstrating robust self-enhancement capabilities. This work establishes a novel training paradigm for GUI agents, shifting from isolated optimization to an integrated, self-driven co-evolution approach.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 36 of its 49 references that are in the map. First posted on arXiv 13 Nov 2025.

Builds on

36 of 49 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning18 May
  2. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
  3. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
  4. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents1 Apr
  5. 2025COLA: A Scalable Multi-Agent Framework For Windows UI Task Automation12 Mar
  6. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  7. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection8 Jan
  8. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  9. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  10. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agent9 Feb
  11. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan

Other references · 25

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  6. 2024DeepSeek-V3 Technical Reportcited by 724
  7. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  8. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  9. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  10. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  11. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  12. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
13 more
  1. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  2. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
  3. 2024OmniParser for Pure Vision Based GUI Agentcited by 193
  4. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  5. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  6. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 123
  7. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
  8. 2025WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Modelcited by 56
  9. 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
  10. 2025Learning to Contextualize Web Pages for Enhanced Decision Making by LLM Agentscited by 16
  11. 2025LiteWebAgent: The Open-Source Suite for VLM-Based Web-Agent Applicationscited by 14
  12. 2025WinClick: GUI Grounding with Multimodal Large Language Modelscited by 11
  13. 2025Importance-Aware Data Selection for Efficient LLM Instruction Tuningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution
  2. 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
  3. 2025CRAFT-GUI: Curriculum-Reinforced Agent For GUI Tasks
  4. 2024AutoGLM: Autonomous Foundation Agents for GUIs
  5. 2025UItron: Foundational GUI Agent with Advanced Perception and Planning
  6. 2026Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation
  7. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
  8. 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience