mlsanityData to 7 Oct 2026

KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill

arXiv 2607.12625 ↗First posted 14 Jul 2026cs.CL cs.CVcited by 1

On the map: Synthetic data and self-improvement for LLMs → Self-evolving LLM agents

Abstract (arXiv)

OpenClaw has emerged as a leading agent framework for complex task automation, yet it faces insufficient cross-platform GUI interaction support and a well-built self-evolution mechanism. These flaws limit its adaptation to diverse device ecosystems and prevent performance improvements through continuous learning from execution experience. To resolve these issues, we propose the Know Deeply, Act Perfectly paradigm for personal assistants, which holds that accumulated user interaction and task-running experience directly improve execution accuracy and efficiency, unifying cognitive comprehension and operational execution. Based on this paradigm, we introduce KnowAct-GUIClaw, a novel Know-Route-Act-Reflect framework designed to address OpenClaw's GUI manipulation deficits and break through its cross-platform and recursive self-improvement constraints. First, the host agent leverages accumulated interaction experience and task-relevant knowledge for long-horizon task decomposition and allocation (Know). Second, a pluggable GUI subagent with an experience-attributable memory system (Know) and self-evolving skill library (Act), enabling seamless cross-platform migration and fast-path integration. Especially, this framework continuously stores user profiles and feedback to improve the accuracy of task decomposition and tool calls. Extensive experiments across Android, iOS, HarmonyOS and Windows show that KnowAct-GUIClaw achieves superior efficiency, accuracy and cross-platform adaptability. Especially, the GUIClaw with open-source Kimi-2.6 models achieves the best performance (64.1%) on the long-horizon MobileWorld benchmark, beating all agentical frameworks and closed-source agentical models, e.g., Seed-2.0-Pro and GPT-5.5. Additionally, the knowledgeable memory and execution skills supported by our framework are transferable across diverse base models, improving by 8.5% with Kimi-2.6.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Self-evolving LLM agents, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 24 of its 55 references that are in the map. First posted on arXiv 14 Jul 2026.

Builds on

24 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvement12 Feb

Other references · 23

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  3. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  4. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  5. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  6. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  7. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  8. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  9. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  10. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  11. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  12. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
11 more
  1. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  2. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
  3. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environmentscited by 69
  4. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontierscited by 60
  5. 2026CUA-Skill: Develop Skills for Computer Using Agentcited by 32
  6. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 20
  7. 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agentscited by 13
  8. 2018Sentence Simplification with Memory-Augmented Neural Networkscited by 10
  9. 2026WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environmentscited by 9
  10. 2025MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agentscited by 8
  11. 2026MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimizationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human
  2. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
  3. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution
  4. 2026Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI Agents
  5. 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration
  6. 2026AppAgent-Claw: CLI Is All You Need for GUI Automation ↗
  7. 2026Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents
  8. 2026Do GUI Agents Know When Not to Act? Enabling Conflict-Aware Termination for Multimodal GUI Agents