Data to 7 Oct 2026

ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay

arXiv 2505.16282 ↗First posted 22 May 2025cs.CVcited by 75

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Training large language models (LLMs) as interactive agents for controlling graphical user interfaces (GUIs) presents a unique challenge to optimize long-horizon action sequences with multimodal feedback from complex environments. While recent works have advanced multi-turn reinforcement learning (RL) for reasoning and tool-using capabilities in LLMs, their application to GUI-based agents remains relatively underexplored due to the difficulty of sparse rewards, delayed feedback, and high rollout costs. In this paper, we investigate end-to-end policy optimization for vision-language-based GUI agents with the aim of improving performance on complex, long-horizon computer tasks. We propose Agentic Replay Policy Optimization (ARPO), an end-to-end RL approach that augments Group Relative Policy Optimization (GRPO) with a replay buffer to reuse the successful experience across training iterations. To further stabilize the training process, we propose a task selection strategy that filters tasks based on baseline agent performance, allowing the agent to focus on learning from informative interactions. Additionally, we compare ARPO with offline preference optimization approaches, highlighting the advantages of policy-based methods in GUI environments. Experiments on the OSWorld benchmark demonstrate that ARPO achieves competitive results, establishing a new performance baseline for LLM-based GUI agents trained via reinforcement learning. Our findings underscore the effectiveness of reinforcement learning for training multi-turn, vision-language GUI agents capable of managing complex real-world UI interactions. Codes and models:https://github.com/dvlab-research/ARPO.git.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 29 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

24 of 29 references.

Closest ancestors · 9

Same direction, newest first

Other references · 15

2017Proximal Policy Optimization Algorithmscited by 14,441
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
2025Qwen2.5-VL Technical Reportcited by 5,418
2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
2025ToolRL: Reward is All Tool Learning Needscited by 342
2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271

3 more

2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 234
2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 31; 2026: 44.

75 papers.

Sort:

By area:

2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
2025Memp: Exploring Agent Procedural Memorycited by 88
2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 59
2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 58
2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 51
2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 49
2025ExGRPO: Learning to Reason from Experiencecited by 49
2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
2025SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agentcited by 44
2025ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluationcited by 35

38 more

2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
2026Code2World: A GUI World Model via Renderable Code Generationcited by 26
2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 25
2025Nudging the Boundaries of LLM Reasoningcited by 25
2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curationcited by 24
2025Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agentscited by 23
2026ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Trainingcited by 23
2025Beyond Turn Limits: Training Deep Search Agents with Dynamic Context Windowcited by 19
2025Mobile-R1: Towards Interactive Capability for VLM-Based Mobile Agent via Systematic Trainingcited by 17
2025Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoningcited by 17
2025GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoningcited by 12
2025WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learningcited by 11
2025STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimizationcited by 11
2025AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoningcited by 11
2026UI-Mem: Self-Evolving Experience Memory for Online Reinforcement Learning in Mobile GUI Agentscited by 11
2026Adaptive Milestone Reward for GUI Agentscited by 11
2026ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agentscited by 10
2025Reinforcement Learning for Large Model: A Surveycited by 8
2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
2025Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalizationcited by 8
2026Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoningcited by 8
2026Efficient RL Training for LLMs with Experience Replaycited by 8
2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoningcited by 7
2025CRAFT-GUI: Curriculum-Reinforced Agent For GUI Taskscited by 6
2025Interact-RAG: Reason and Interact with the Corpus, Beyond Black-Box Retrievalcited by 6
2026PhGPO: Pheromone-Guided Policy Optimization for Long-Horizon Tool Planningcited by 6
2026Generalization in Online Reinforcement Learning for Mobile Agentscited by 6
2026Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actionscited by 6
2026Learning from the Irrecoverable: Error-Localized Policy Optimization for Tool-Integrated LLM Reasoningcited by 5
2026On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Lengthcited by 5
2026Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretrainingcited by 5
2026Beyond Experience Retrieval: Learning to Generate Utility-Optimized Structured Experience for Frozen LLMscited by 4
2026AEGPO: Adaptive Entropy-Guided Policy Optimization for Diffusion Modelscited by 4
2026Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problemscited by 4
2026UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimizationcited by 4
2026DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replaycited by 3
2026Turing Test on Screen: A Benchmark for Mobile GUI Agent Humanizationcited by 3
2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way