# KG-RAG: Enhancing GUI Agent Decision-Making via Knowledge Graph-Driven Retrieval-Augmented Generation

arXiv: https://arxiv.org/abs/2509.00366 · HTML: https://mlsanity.com/paper/2509.00366/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation.
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 30 Aug 2025.

## Placed in

- Retrieval-augmented generation / [Agentic RAG, agents and memory](https://mlsanity.com/rag/agentic-rag-agents-and-memory/)
- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 11 of 11)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [AppAgent: Multimodal Agents as Smartphone Users](https://mlsanity.com/paper/2312.13771/) (2023)
- [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://mlsanity.com/paper/2407.19669/) (2024)
- [AutoDroid: LLM-powered Task Automation in Android](https://mlsanity.com/paper/2308.15272/) (2023)
- [Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs](https://mlsanity.com/paper/2404.05719/) (2024)
- [VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling](https://mlsanity.com/paper/2112.05692/) (2021)
- [Autonomous Large Language Model Agents Enabling Intent-Driven Mobile GUI Testing](https://mlsanity.com/paper/2311.08649/) (2023)

## Built on by (most cited first, 9 of 9)

- [CUA-Skill: Develop Skills for Computer Using Agent](https://mlsanity.com/paper/2601.21123/) (2026)
- [Continual GUI Agents](https://mlsanity.com/paper/2601.20732/) (2026)
- [Executable Agentic Memory for GUI Agent](https://mlsanity.com/paper/2605.12294/) (2026)
- [CoVe: Training Interactive Tool-Use Agents via Constraint-Guided Verification](https://mlsanity.com/paper/2603.01940/) (2026)
- [GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation](https://mlsanity.com/paper/2603.26266/) (2026)
- [GUI-AC: Enhancing Continual Learning in GUI Agents](https://mlsanity.com/paper/2606.10522/) (2026)
- [Beyond Binary: Reframing GUI Critique as Continuous Semantic Alignment](https://mlsanity.com/paper/2605.14311/) (2026)
- [SynWeaver: Website-Prior Task and Trajectory Co-Synthesis for Web Agents](https://mlsanity.com/paper/2608.12429/) (2026)
- [GraMRAG: Orchestrating Multi-Agent Multi-Step Reasoning via Graph Memory with Reinforcement Learning](https://mlsanity.com/paper/2609.14066/) (2026)

## Abstract (arXiv)

Despite recent progress, Graphic User Interface (GUI) agents powered by Large Language Models (LLMs) struggle with complex mobile tasks due to limited app-specific knowledge. While UI Transition Graphs (UTGs) offer structured navigation representations, they are underutilized due to poor extraction and inefficient integration. We introduce KG-RAG, a Knowledge Graph-driven Retrieval-Augmented Generation framework that transforms fragmented UTGs into structured vector databases for efficient real-time retrieval. By leveraging an intent-guided LLM search method, KG-RAG generates actionable navigation paths, enhancing agent decision-making. Experiments across diverse mobile apps show that KG-RAG outperforms existing methods, achieving a 75.8% success rate (8.9% improvement over AutoDroid), 84.6% decision accuracy (8.1% improvement), and reducing average task steps from 4.5 to 4.1. Additionally, we present KG-Android-Bench and KG-Harmony-Bench, two benchmarks tailored to the Chinese mobile ecosystem for future research. Finally, KG-RAG transfers to web/desktop (+40% SR on Weibo-web; +20% on QQ Music-desktop), and a UTG cost ablation shows accuracy saturates at ~4h per complex app, enabling practical deployment trade-offs.