# ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaboration

arXiv: https://arxiv.org/abs/2509.21823 · HTML: https://mlsanity.com/paper/2509.21823/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 31 of its 35 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 31)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024)
- [WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning](https://mlsanity.com/paper/2411.02337/) (2024)
- [GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents](https://mlsanity.com/paper/2504.10458/) (2025)
- [Inference-Time Scaling for Generalist Reward Modeling](https://mlsanity.com/paper/2504.02495/) (2025)
- [Agent-as-a-Judge: Evaluate Agents with Agents](https://mlsanity.com/paper/2410.10934/) (2024)
- [UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2509.02544/) (2025)
- [Mobile-Agent-v3: Fundamental Agents for GUI Automation](https://mlsanity.com/paper/2508.15144/) (2025)
- [InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners](https://mlsanity.com/paper/2504.14239/) (2025)
- [AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents](https://mlsanity.com/paper/2410.24024/) (2024)
- [UI-Venus Technical Report: Building High-performance UI Agents with RFT](https://mlsanity.com/paper/2508.10833/) (2025)
- [On Designing Effective RL Reward at Training Time for LLM Reasoning](https://mlsanity.com/paper/2410.15115/) (2024)
- [Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge](https://mlsanity.com/paper/2506.21506/) (2025)
- [Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation](https://mlsanity.com/paper/2402.11443/) (2024)
- [DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents](https://mlsanity.com/paper/2410.14803/) (2024)
- [MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents](https://mlsanity.com/paper/2406.08184/) (2024)
- [GUI-G2: Gaussian Reward Modeling for GUI Grounding](https://mlsanity.com/paper/2507.15846/) (2025)
- [ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents](https://mlsanity.com/paper/2508.14040/) (2025)
- [MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents](https://mlsanity.com/paper/2509.18119/) (2025)
- [Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deployment](https://mlsanity.com/paper/2503.15937/) (2025)
- [Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation](https://mlsanity.com/paper/2502.14948/) (2025)
- [LLM-as-a-Judge & Reward Model: What They Can and Cannot Do](https://mlsanity.com/paper/2409.11239/) (2024)
- [StepWiser: Stepwise Generative Judges for Wiser Reasoning](https://mlsanity.com/paper/2508.19229/) (2025)
- [When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs](https://mlsanity.com/paper/2508.02994/) (2025)
- [Evolving in Tasks: Empowering the Multi-modality Large Language Model as the Computer Use Agent](https://mlsanity.com/paper/2508.04037/) (2025)
- [AndroidGen: Building an Android Language Agent under Data Scarcity](https://mlsanity.com/paper/2504.19298/) (2025)
- [MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operation](https://mlsanity.com/paper/2507.16853/) (2025)

## Built on by (most cited first, 11 of 11)

- [Adaptive Milestone Reward for GUI Agents](https://mlsanity.com/paper/2602.11524/) (2026)
- [Agentic Reward Modeling: Verifying GUI Agent via Progressive Trajectory-Grounded Interaction](https://mlsanity.com/paper/2602.00575/) (2026)
- [GUI Agents with Reinforcement Learning: Toward Digital Inhabitants](https://mlsanity.com/paper/2604.27955/) (2026)
- [ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis](https://mlsanity.com/paper/2605.25160/) (2026)
- [StainFlow: Entity-Stain Tracking and Evidence Linking for Process Rewards in GUI Agents](https://mlsanity.com/paper/2606.07027/) (2026)
- [SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents](https://mlsanity.com/paper/2607.23263/) (2026)
- [Benchmarking LLM Judges for Mobile Agent Evaluation](https://mlsanity.com/paper/2608.11434/) (2026)
- [Task-Adaptive Rubrics for GUI Reward Modeling](https://mlsanity.com/paper/2608.24174/) (2026)
- [ARS: Agentic Reward System for Robot Learning](https://mlsanity.com/paper/2609.34484/) (2026)
- [SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use Safety](https://mlsanity.com/paper/2609.36201/) (2026)
- [Adaptive-Group Process Rewards for GUI Agents](https://mlsanity.com/paper/2610.17710/) (2026)

## Abstract (arXiv)

Reward is critical to the evaluation and training of large language models (LLMs). However, existing rule-based or model-based reward methods struggle to generalize to GUI agents, where access to ground-truth trajectories or application databases is often unavailable, and static trajectory-based LLM-as-a-Judge approaches suffer from limited accuracy. To address these challenges, we propose ProRe, a proactive reward system that leverages a general-purpose reasoner and domain-specific evaluator agents (actors). The reasoner schedules targeted state probing tasks, which the evaluator agents then execute by actively interacting with the environment to collect additional observations. This enables the reasoner to assign more accurate and verifiable rewards to GUI agents. Empirical results on over 3K trajectories demonstrate that ProRe improves reward accuracy and F1 score by up to 5.3% and 19.4%, respectively. Furthermore, integrating ProRe with state-of-the-art policy agents yields a success rate improvement of up to 22.4%. The source code is available at https://github.com/V-Droid-Agent/ProRe.