# Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction

arXiv: https://arxiv.org/abs/2610.12984 · HTML: https://mlsanity.com/paper/2610.12984/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 32 of its 55 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 30 of 32)

- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models](https://mlsanity.com/paper/2510.04618/) (2025)
- [SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning](https://mlsanity.com/paper/2602.08234/) (2026)
- [SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills](https://mlsanity.com/paper/2504.07079/) (2025)
- [Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills](https://mlsanity.com/paper/2603.25158/) (2026)
- [Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory](https://mlsanity.com/paper/2504.07952/) (2025)
- [Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents](https://mlsanity.com/paper/2504.00906/) (2025)
- [MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents](https://mlsanity.com/paper/2602.02474/) (2026)
- [AutoSkill: Experience-Driven Lifelong Learning via Skill Self-Evolution](https://mlsanity.com/paper/2603.01145/) (2026)
- [Inducing Programmatic Skills for Agentic Tasks](https://mlsanity.com/paper/2504.06821/) (2025)
- [GUI Agents: A Survey](https://mlsanity.com/paper/2412.13501/) (2024)
- [UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction](https://mlsanity.com/paper/2503.15661/) (2025)
- [DrVideo: Document Retrieval Based Long Video Understanding](https://mlsanity.com/paper/2406.12846/) (2024)
- [A Survey on (M)LLM-Based GUI Agents](https://mlsanity.com/paper/2504.13865/) (2025)
- [VCA: Video Curious Agent for Long Video Understanding](https://mlsanity.com/paper/2412.10471/) (2024)
- [OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents](https://mlsanity.com/paper/2506.16042/) (2025)
- [Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understanding](https://mlsanity.com/paper/2512.05774/) (2025)
- [Grounding Computer Use Agents on Human Demonstrations](https://mlsanity.com/paper/2511.07332/) (2025)
- [LensWalk: Agentic Video Understanding by Planning How You See in Videos](https://mlsanity.com/paper/2603.24558/) (2026)
- [MMSkills: Towards Multimodal Skills for General Visual Agents](https://mlsanity.com/paper/2605.13527/) (2026)
- [ShowUI-Aloha: Human-Taught GUI Agent](https://mlsanity.com/paper/2601.07181/) (2026)
- [MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents](https://mlsanity.com/paper/2606.16748/) (2026)
- [CUADebug: Diagnosing and Repairing Computer-Use Agent Failures](https://mlsanity.com/paper/2608.02643/) (2026)
- [SkillForge: Evolving Verifiable Skills for Reinforcement Learning Agents](https://mlsanity.com/paper/2608.24747/) (2026)
- [Demo2Tutorial: From Human Experience to Multimodal Software Tutorials](https://mlsanity.com/paper/2606.03951/) (2026)
- [Scalable Behaviour Cloning on Browser Using via Skill Distillation](https://mlsanity.com/paper/2606.32014/) (2026)
- [EvoClawBench: Can Agents Learn Reusable Skills from Their Own Runs?](https://mlsanity.com/paper/2607.09711/) (2026)
- [Instruction Agent: Enhancing Agent with Expert Demonstration](https://mlsanity.com/paper/2509.07098/) (2025)
- [GPA: Learning GUI Process Automation from Demonstrations](https://mlsanity.com/paper/2604.01676/) (2026)

## Abstract (arXiv)

Computer-use agents can perform increasingly complex GUI tasks, yet unfamiliar software and workflows requiring human domain knowledge remain challenging. Human demonstrations convey procedural knowledge through the same GUI interaction space, but reuse requires interpreting actions in context and separating workflow rules from task-specific details. We thus introduce Work2Skill, a framework that captures human GUI experience for reusable agent skill induction. WorkRecorder provides the framework's infrastructure for collecting this experience, aligning screen recordings, mouse and keyboard events, and available accessibility-tree information. Using WorkRecorder, we construct WorkDemo-350, the first synchronized multimodal dataset of human desktop GUI demonstrations for reusable agent skill induction and downstream evaluation. Its 350 demonstrations are screened for completeness and task success. We then propose an agentic skill-induction method that inspects evidence on demand to produce reusable skills capturing workflow rules, step dependencies, and completion checks. When the same four execution models are equipped with skills induced by each method under shared execution settings, Work2Skill achieves the highest scores and success rates. Mean score improvements over the corresponding No Skill baseline are 4.67--10.54 points on MyPCBench Bounded Action and 3.11--7.50 points on a 286-task subset of OSWorld-Verified. Our code and data are available at https://github.com/china-qijizhifeng/Work2Skill.