mlsanityData to 14 Oct 2026

What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States

arXiv 2606.31612 ↗First posted 30 Jun 2026cs.CVcited by 5

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Mobile GUI agents increasingly face long-horizon tasks that require reading, updating, and reusing task-relevant data across pages and applications. Existing methods treat memory largely as passive storage, where past observations are accumulated and retrieved when needed. Yet retrieving a value does not reveal its current role in the workflow. The agent must still infer from accumulated records whether the value should be used now, has already been used, or must wait for a later dependency. This implicit reconstruction becomes unreliable in long trajectories with repeated values, distractors, and outdated states, causing repeated or missed operations. To address this, we propose Active Task Driving Memory (ATMem), which shifts GUI-agent memory from passive storage to an actively maintained execution state. ATMem maintains task-relevant information as a continually updated execution state that links each value to its role and current status, enabling action selection based on the current workflow state. While supervised fine-tuning enables the agent to construct ATMem, it does not teach when ATMem is beneficial. We therefore introduce STR-GRPO, an online reinforcement learning method that encourages selective use of ATMem based on its contribution to task completion. STR-GRPO contrasts memory-on and memory-off rollouts to estimate when memory use improves execution, while memory-cost-aware reward discourages costly memory usage that does not improve execution. To evaluate whether agents can complete all in-scope work while avoiding out-of-scope actions, we build a challenging mobile benchmark. From a list of near identical entries, agents must act on every entry that satisfies the instruction and reject entries that violate its constraints. We further introduce App-Level Progress and Scope-Aware F1 to measure these two dimensions separately.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 41 of its 49 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

41 of 49 references.

Closest ancestors · 28

Same direction, newest first

  1. 2026AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents19 Mar
  2. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents15 Feb
  3. 2026UI-Venus-1.5 Technical Report9 Feb
  4. 2026UI-Mem: Self-Evolving Experience Memory for Online Reinforcement Learning in Mobile GUI Agents5 Feb
  5. 2026MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolution27 Jan
  6. 2026GraphPilot: GUI Task Automation with One-Step LLM Reasoning Powered by Knowledge Graph24 Jan
  7. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec
  8. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments22 Dec
  9. 2025EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Exploration22 Dec
  10. 2025Step-GUI Technical Report17 Dec
  11. 2025AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Management11 Dec
  12. 2025ProBench: Benchmarking GUI Agents with Accurate Process Information12 Nov
16 more
  1. 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interaction28 Oct
  2. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning23 Oct
  3. 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents10 Sep
  4. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2 Sep
  5. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation21 Aug
  6. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFT14 Aug
  7. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment8 Jul
  8. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  9. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks20 Jan
  10. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents31 Oct
  11. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation19 Oct
  12. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactions5 Aug
  13. 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents1 Jul
  14. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration3 Jun
  15. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
  16. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec

Other references · 13

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2025Qwen3-VL Technical Reportcited by 2,793
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  5. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  6. 2023Cognitive Architectures for Language Agentscited by 429
  7. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  8. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 184
  9. 2023Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automationcited by 58
  10. 2026MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environmentscited by 18
  11. 2024MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automationcited by 14
  12. 2026Hybrid Self-evolving Structured Memory for GUI Agentscited by 13
1 more
  1. 2026SkillDroid: Compile Once, Reuse Forevercited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026DeltaReplay: Task-Relative Memory Reuse for Mobile GUI Agents
  2. 2026A Task-State Representation for Long-Horizon Mobile GUI Agents
  3. 2026STAMP: Training Explicit Memory for Mobile GUI Agents in Controllable and Scalable Virtual Environments
  4. 2026Mem-W: Latent Memory-Native GUI Agents
  5. 2025Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation
  6. 2026Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization
  7. 2026ActionEngine: From Reactive to Programmatic GUI Agents via State Machine Memory
  8. 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents