# GraphPilot: GUI Task Automation with One-Step LLM Reasoning Powered by Knowledge Graph

arXiv: https://arxiv.org/abs/2601.17418 · HTML: https://mlsanity.com/paper/2601.17418/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 7 of its 22 references that are in the map.
- First posted on arXiv 24 Jan 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 7 of 7)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [DroidBot-GPT: GPT-powered UI Automation for Android](https://mlsanity.com/paper/2304.07061/) (2023)
- [AppAgentX: Evolving GUI Agents as Proficient Smartphone Users](https://mlsanity.com/paper/2503.02268/) (2025)
- [MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices](https://mlsanity.com/paper/2407.03913/) (2024)
- [GUI Action Narrator: Where and When Did That Action Take Place?](https://mlsanity.com/paper/2406.13719/) (2024)
- [MMAC-Copilot: Multi-modal Agent Collaboration Operating Copilot](https://mlsanity.com/paper/2404.18074/) (2024)
- [Seeing is Believing: Vision-driven Non-crash Functional Bug Detection for Mobile Apps](https://mlsanity.com/paper/2407.03037/) (2024)

## Built on by (most cited first, 6 of 6)

- [What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States](https://mlsanity.com/paper/2606.31612/) (2026)
- [UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents](https://mlsanity.com/paper/2605.29534/) (2026)
- [Integrating Graphs, Large Language Models, and Agents: Reasoning and Retrieval](https://mlsanity.com/paper/2604.15951/) (2026)
- [Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability](https://mlsanity.com/paper/2605.07110/) (2026)
- [When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents](https://mlsanity.com/paper/2609.16732/) (2026)
- [How GUI Agents Fail: GraceBench for Execution Stability Diagnosis](https://mlsanity.com/paper/2610.19900/) (2026)

## Abstract (arXiv)

Mobile graphical user interface (GUI) agents are designed to automate everyday tasks on smartphones. Recent advances in large language models (LLMs) have significantly enhanced the capabilities of mobile GUI agents. However, most LLM-powered mobile GUI agents operate in stepwise query-act loops, which incur high latency due to repeated LLM queries. We present GraphPilot, a mobile GUI agent that leverages knowledge graphs of the target apps to complete user tasks in almost one LLM query. GraphPilot operates in two complementary phases to enable efficient and reliable LLM-powered GUI task automation. In the offline phase, it explores target apps, records and analyzes interaction history, and constructs an app-specific knowledge graph that encodes functions of pages and elements as well as transition rules for each app. In the online phase, given an app and a user task, it leverages the knowledge graph of the given app to guide the reasoning process of LLM. When the reasoning process encounters uncertainty, GraphPilot dynamically requests the HTML representation of the current interface to refine subsequent reasoning. Finally, a validator checks the generated sequence of actions against the transition rules in the knowledge graph, performing iterative corrections to ensure it is valid. The structured, informative information in the knowledge graph allows the LLM to plan the complete sequence of actions required to complete the user task. On the DroidTask benchmark, GraphPilot improves task completion rate over Mind2Web and AutoDroid, while substantially reducing latency and the number of LLM queries.