# Log2Plan: An Adaptive GUI Automation Framework Integrated with Task Mining Approach

arXiv: https://arxiv.org/abs/2509.22137 · HTML: https://mlsanity.com/paper/2509.22137/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 45 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Builds on (most cited first, 21 of 21)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [UFO: A UI-Focused Agent for Windows OS Interaction](https://mlsanity.com/paper/2402.07939/) (2024)
- [ScreenAgent: A Vision Language Model-driven Computer Control Agent](https://mlsanity.com/paper/2402.07945/) (2024)
- [Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus](https://mlsanity.com/paper/2209.14927/) (2022)
- [A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility](https://mlsanity.com/paper/2202.02312/) (2022)
- [Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms](https://mlsanity.com/paper/2410.18967/) (2024)
- [Hierarchical Prompting Assists Large Language Model on Web Navigation](https://mlsanity.com/paper/2305.14257/) (2023)
- [VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning](https://mlsanity.com/paper/2312.11190/) (2023)
- [Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations](https://mlsanity.com/paper/1909.00031/) (2019)
- [Dynamic Planning for LLM-based Graphical User Interface Automation](https://mlsanity.com/paper/2410.00467/) (2024)
- [ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations](https://mlsanity.com/paper/2310.04869/) (2023)
- [TaskLAMA: Probing the Complex Task Understanding of Language Models](https://mlsanity.com/paper/2308.15299/) (2023)
- [Automatic Macro Mining from Interaction Traces at Scale](https://mlsanity.com/paper/2310.07023/) (2023)
- [VASTA: A Vision and Language-assisted Smartphone Task Automation System](https://mlsanity.com/paper/1911.01474/) (2019)
- [Skill Learning Using Process Mining for Large Language Model Plan Generation](https://mlsanity.com/paper/2410.12870/) (2024)

## Abstract (arXiv)

GUI task automation streamlines repetitive tasks, but existing LLM or VLM-based planner-executor agents suffer from brittle generalization, high latency, and limited long-horizon coherence. Their reliance on single-shot reasoning or static plans makes them fragile under UI changes or complex tasks. Log2Plan addresses these limitations by combining a structured two-level planning framework with a task mining approach over user behavior logs, enabling robust and adaptable GUI automation. Log2Plan constructs high-level plans by mapping user commands to a structured task dictionary, enabling consistent and generalizable automation. To support personalization and reuse, it employs a task mining approach from user behavior logs that identifies user-specific patterns. These high-level plans are then grounded into low-level action sequences by interpreting real-time GUI context, ensuring robust execution across varying interfaces. We evaluated Log2Plan on 200 real-world tasks, demonstrating significant improvements in task success rate and execution time. Notably, it maintains over 60.0% success rate even on long-horizon task sequences, highlighting its robustness in complex, multi-step workflows.