mlsanityData to 7 Oct 2026

GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices

arXiv 2406.08451 ↗First posted 12 Jun 2024cs.CVcited by 190

Not on the map. Why?

Cited by 146 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Autonomous Graphical User Interface (GUI) navigation agents can enhance user experience in communication, entertainment, and productivity by streamlining workflows and reducing manual intervention. However, prior GUI agents often trained with datasets comprising tasks that can be completed within a single app, leading to poor performance in cross-app navigation. To address this problem, we present GUIOdyssey, a comprehensive dataset for cross-app mobile GUI navigation. GUIOdyssey comprises 8,334 episodes with an average of 15.3 steps per episode, covering 6 mobile devices, 212 distinct apps, and 1,357 app combinations. Each step is enriched with detailed semantic reasoning annotations, which aid the model in building cognitive processes and enhancing its reasoning abilities for complex cross-app tasks. Building on GUIOdyssey, we develop OdysseyAgent, an exploratory multimodal agent for long-step cross-app navigation equipped with a history resampler module that efficiently attends to historical screenshot tokens, balancing performance and inference speed. Extensive experiments conducted in both in-domain and out-of-domain scenarios validate the effectiveness of our approach. Moreover, we demonstrate that historial information involving actions, screenshots and context in our dataset can significantly enhances OdysseyAgent's performance on complex cross-app tasks.

Key facts. Cited by 190 papers in the map (data through 7 Oct 2026). Builds on 50 of its 73 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

50 of 73 references.

References · 50

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
38 more
  1. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  5. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  6. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  7. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  8. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  9. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  10. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  11. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  12. 2019Scene Text Visual Question Answeringcited by 327
  13. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  14. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  15. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  16. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  17. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  18. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  19. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  20. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  21. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  22. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  23. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  24. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  25. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  26. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  27. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  28. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  29. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  30. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  31. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  32. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  33. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  34. 2024Understanding the Weakness of Large Language Model Agents within a Complex Android Environmentcited by 60
  35. 2023Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generationcited by 38
  36. 2021Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environmentscited by 28
  37. 2022UGIF: UI Grounded Instruction Followingcited by 25
  38. 2024Self-Refine Instruction-Tuning for Aligning Reasoning in Language Modelscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 83; 2026: 92.

190 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  4. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  5. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  6. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 240
  7. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  8. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  9. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
  10. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  11. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  12. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
38 more
  1. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  2. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
  3. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 79
  4. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  5. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  6. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61
  7. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  8. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  9. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  10. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  11. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  12. 2026UI-Venus-1.5 Technical Reportcited by 39
  13. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  14. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 33
  15. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  16. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  17. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  18. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generationcited by 30
  19. 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
  20. 2025OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agentscited by 25
  21. 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 25
  22. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 22
  23. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  24. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  25. 2025Auto-scaling Continuous Memory for GUI Agentcited by 19
  26. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  27. 2025SpiritSight Agent: Advanced GUI Agent with One Lookcited by 18
  28. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  29. 2025Efficient Agent Training for Computer Usecited by 17
  30. 2025Mobile-R1: Towards Interactive Capability for VLM-Based Mobile Agent via Systematic Trainingcited by 17
  31. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  32. 2026PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Recordscited by 16
  33. 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Trainingcited by 16
  34. 2025GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomaliescited by 15
  35. 2025VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agentscited by 15
  36. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 14
  37. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 14
  38. 2026Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correctioncited by 14

50 most cited of 190 All 190 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration
  2. 2025GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent
  3. 2026UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents
  4. 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
  5. 2026MobileExplorer: Accelerating On-Device Inference for Mobile GUI Agents via Online Exploration
  6. 2024MobileViews: A Million-scale and Diverse Mobile GUI Dataset
  7. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation
  8. 2024Falcon-UI: Understanding GUI Before Following User Instructions