mlsanityData to 13 Oct 2026

IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents

arXiv 2604.05157 ↗First posted 6 Apr 2026cs.AIcited by 1

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Computer-Use Agents (CUAs) leverage large language models to execute GUI operations on desktop environments, yet they generate actions without evaluating action quality, leading to irreversible errors that cascade through subsequent steps. We propose IntentScore, a plan-aware reward model that learns to score candidate actions from 398K offline GUI interaction steps spanning three operating systems. IntentScore trains with two complementary objectives: contrastive alignment for state-action relevance and margin ranking for action correctness. Architecturally, it embeds each candidate's planning intent in the action encoder, enabling discrimination between candidates with similar actions but different rationales. IntentScore achieves 97.5% pairwise discrimination accuracy on held-out evaluation. Deployed as a re-ranker for Agent S3 on OSWorld, an environment entirely unseen during training, IntentScore improves task success rate by 6.9 points, demonstrating that reward estimation learned from heterogeneous offline trajectories generalizes to unseen agents and task distributions.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 45 of its 51 references that are in the map. First posted on arXiv 6 Apr 2026.

Builds on

45 of 51 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data18 Sep
  2. 2025OpenCUA: Open Foundations for Computer-Use Agents12 Aug
  3. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents1 Apr
  4. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directions27 Jan
  5. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  6. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human10 Oct
  7. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration3 Jun
  8. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  9. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,367
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,068
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,352
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,953
  7. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,676
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,054
  9. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 976
  10. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 551
  11. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 456
  12. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixelscited by 256
24 more
  1. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  2. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  3. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  4. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
  5. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  6. 2025A Survey of In-Context Reinforcement Learningcited by 36
  7. 2022BALF: Simple and Efficient Blur Aware Local Feature Detectorcited by 34
  8. 2026CUA-Skill: Develop Skills for Computer Using Agentcited by 33
  9. 2026ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learningcited by 29
  10. 2025AgentRM: Enhancing Agent Generalization with Reward Modelingcited by 25
  11. 2026Computer-Using World Modelcited by 16
  12. 2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
  13. 2026Advances in Global Solvers for 3D Visioncited by 13
  14. 2025Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planningcited by 12
  15. 2026Agent Alpha: Tree Search Unifying Generation, Exploration and Evaluation for Computer-Use Agentscited by 12
  16. 2026When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPOcited by 10
  17. 2025Vintix: Action Model via In-Context Reinforcement Learningcited by 9
  18. 2024Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Modelscited by 8
  19. 2026Agentic Test-Time Scaling for WebAgentscited by 8
  20. 2026Manifold-Constrained Energy-Based Transition Models for Offline Reinforcement Learningcited by 7
  21. 2026Structuring Value Representations via Geometric Coherence in Markov Decision Processescited by 6
  22. 2024Image Matching Filtering and Refinement by Planes and Beyondcited by 5
  23. 2026LiSFC-Search: Lifelong Search for Network SFC Optimization under Non-stationary Driftscited by 5
  24. 2026Knowing When to Ask: Resolving Uncertainty in Human-Robot Joint Planning via Explicit Dialogue and Implicit Intent Cuescited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026IntentCUA: Learning Intent-level Representations for Skill Abstraction and Multi-Agent Planning in Computer-Use Agents
  2. 2026OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
  3. 2026ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories
  4. 2026IntPro: A Proxy Agent for Context-Aware Intent Understanding via Retrieval-conditioned Inference
  5. 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agents
  6. 2026When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents
  7. 2025CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent
  8. 2026MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents via Automated Feedback Reflux