GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
GUI automation faces critical challenges in dynamic environments. MLLMs suffer from two key issues: misinterpreting UI components and outdated knowledge. Traditional fine-tuning methods are costly for app-specific knowledge updates. We propose GUI-explorer, a training-free GUI agent that incorporates two fundamental mechanisms: (1) Autonomous Exploration of Function-aware Trajectory. To comprehensively cover all application functionalities, we design a Function-aware Task Goal Generator that automatically constructs exploration goals by analyzing GUI structural information (e.g., screenshots and activity hierarchies). This enables systematic exploration to collect diverse trajectories. (2) Unsupervised Mining of Transition-aware Knowledge. To establish precise screen-operation logic, we develop a Transition-aware Knowledge Extractor that extracts effective screen-operation logic through unsupervised analysis the state transition of structured interaction triples (observation, action, outcome). This eliminates the need for human involvement in knowledge extraction. With a task success rate of 53.7% on SPA-Bench and 47.4% on AndroidWorld, GUI-explorer shows significant improvements over SOTA agents. It requires no parameter updates for new apps. GUI-explorer is open-sourced and publicly available at https://github.com/JiuTian-VL/GUI-explorer.
Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 45 references that are in the map. First posted on arXiv 22 May 2025.
Builds on
28 of 45 references.
Closest ancestors · 7
Same direction, newest first
- 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control2 Dec
- 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation19 Oct
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
- 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
- 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration3 Jun
- 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
- 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
Other references · 21
- 2023GPT-4 Technical Reportcited by 18,396
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
- 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
- 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 187
- 2023AutoDroid: LLM-powered Task Automation in Androidcited by 184
- 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
- 2023Detecting and Grounding Multi-Modal Media Manipulationcited by 99
9 more
- 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 84
- 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 81
- 2023Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automationcited by 58
- 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 58
- 2023Detecting and Grounding Multi-Modal Media Manipulation and Beyondcited by 56
- 2024VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thoughtcited by 55
- 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
- 2025Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policycited by 25
- 2024Enabling Cost-Effective UI Automation Testing with Retrieval-Based LLMs: A Case Study in WeChatcited by 6
Built on by
Citing papers per year: 2025: 13; 2026: 22.
35 papers.
Sort:
By area:
- 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
- 2026UI-Venus-1.5 Technical Reportcited by 41
- 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
- 2026AgentOCR: Reimagining Agent History via Optical Self-Compressioncited by 22
- 2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skillscited by 19
- 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidancecited by 16
- 2026PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Recordscited by 16
- 2025Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinementcited by 15
- 2025EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Explorationcited by 15
- 2025HiconAgent: History Context-aware Policy Optimization for GUI Agentscited by 13
- 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learningcited by 11
- 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
23 more
- 2026MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimizationcited by 7
- 2026Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actionscited by 6
- 2026ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulationcited by 6
- 2025Mobile-Agent-RAG: Driving Smart Multi-Agent Coordination with Contextual Knowledge Empowerment for Long-Horizon Mobile Automationcited by 5
- 2025MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agentscited by 5
- 2026GUI-GenBench: Evaluating Image Generation Models as Interactive GUI Environmentscited by 5
- 2026PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agentcited by 5
- 2026Executable Agentic Memory for GUI Agentcited by 5
- 2025OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worldscited by 4
- 2025Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbationscited by 3
- 2025D-Artemis: A Deliberative Cognitive Framework for Mobile GUI Multi-Agentscited by 1
- 2026Curiosity Driven Knowledge Retrieval for Mobile Agentscited by 1
- 2026Environment Maps: Structured Environmental Representations for Long-Horizon Agentscited by 1
- 2026DocOS: Towards Proactive Document-Guided Actions in GUI Agentscited by 1
- 2026Software Engineering for and with GUI Agentcited by 1
- 2026SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesiscited by 0
- 2026MobileExplorer: Accelerating On-Device Inference for Mobile GUI Agents via Online Explorationcited by 0
- 2026GUI agent: Guided Exploration of User-Sensitive Screenscited by 0
- 2026GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesiscited by 0
- 2026Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimizationcited by 0
- 2026When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agentscited by 0
- 2026PhoneCLI: From App Interfaces to Callable Commands for Mobile Agentscited by 0
- 2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agentcited by 0
- 2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agent1 Oct 2026
- 2026PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents28 Sep 2026
- 2026When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents15 Sep 2026
- 2026Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization2 Sep 2026
- 2026GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis24 Aug 2026
- 2026Software Engineering for and with GUI Agent10 Aug 2026
- 2026GUI agent: Guided Exploration of User-Sensitive Screens24 Jun 2026
- 2026MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization18 Jun 2026
- 2026MobileExplorer: Accelerating On-Device Inference for Mobile GUI Agents via Online Exploration26 May 2026
- 2026DocOS: Towards Proactive Document-Guided Actions in GUI Agents18 May 2026
- 2026Executable Agentic Memory for GUI Agent12 May 2026
- 2026ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulation6 May 2026
23 more
- 2026Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actions8 Apr 2026
- 2026PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent31 Mar 2026
- 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience25 Mar 2026
- 2026Environment Maps: Structured Environmental Representations for Long-Horizon Agents24 Mar 2026
- 2026UI-Venus-1.5 Technical Report9 Feb 2026
- 2026GUI-GenBench: Evaluating Image Generation Models as Interactive GUI Environments9 Feb 2026
- 2026Curiosity Driven Knowledge Retrieval for Mobile Agents27 Jan 2026
- 2026SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesis26 Jan 2026
- 2026PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Records14 Jan 2026
- 2026AgentOCR: Reimagining Agent History via Optical Self-Compression8 Jan 2026
- 2025EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Exploration22 Dec 2025
- 2025HiconAgent: History Context-aware Policy Optimization for GUI Agents1 Dec 2025
- 2025MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agents30 Nov 2025
- 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidance17 Nov 2025
- 2025Mobile-Agent-RAG: Driving Smart Multi-Agent Coordination with Contextual Knowledge Empowerment for Long-Horizon Mobile Automation15 Nov 2025
- 2025Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbations1 Oct 2025
- 2025D-Artemis: A Deliberative Cognitive Framework for Mobile GUI Multi-Agents26 Sep 2025
- 2025OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds2 Sep 2025
- 2025UItron: Foundational GUI Agent with Advanced Perception and Planning29 Aug 2025
- 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey18 Aug 2025
- 2025Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinement6 Aug 2025
- 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning10 Jul 2025
- 2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills12 Jun 2025
- 2026UI-Venus-1.5 Technical Reportcited by 41
- 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
- 2026AgentOCR: Reimagining Agent History via Optical Self-Compressioncited by 22
- 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidancecited by 16
- 2026PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Recordscited by 16
- 2025Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinementcited by 15
- 2025HiconAgent: History Context-aware Policy Optimization for GUI Agentscited by 13
- 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
- 2026MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimizationcited by 7
- 2026Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actionscited by 6
- 2025Mobile-Agent-RAG: Driving Smart Multi-Agent Coordination with Contextual Knowledge Empowerment for Long-Horizon Mobile Automationcited by 5
- 2026PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agentcited by 5
12 most cited of 25 All 25 (JSON)
- 2026PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents28 Sep 2026
- 2026When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents15 Sep 2026
- 2026Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization2 Sep 2026
- 2026GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis24 Aug 2026
- 2026Software Engineering for and with GUI Agent10 Aug 2026
- 2026GUI agent: Guided Exploration of User-Sensitive Screens24 Jun 2026
- 2026MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization18 Jun 2026
- 2026MobileExplorer: Accelerating On-Device Inference for Mobile GUI Agents via Online Exploration26 May 2026
- 2026DocOS: Towards Proactive Document-Guided Actions in GUI Agents18 May 2026
- 2026Executable Agentic Memory for GUI Agent12 May 2026
- 2026Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actions8 Apr 2026
- 2026PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent31 Mar 2026
12 newest of 25 All 25 (JSON)
Similar, not cited
Close in wording, no citation either way
- 2026UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents
- 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration
- 2026SEE: Structure-aware Exploring & Exploiting for Long-horizon GUI Agent Trajectory Synthesis
- 2026Learning Reliable GUI Agents under Imperfect Priors
- 2024Falcon-UI: Understanding GUI Before Following User Instructions
- 2025AUTO-Explorer: Automated Data Collection for GUI Agent
- 2025GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoning
- 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices