mlsanityData to 14 Oct 2026

META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI

arXiv 2205.11029 ↗First posted 23 May 2022cs.CL cs.AIcited by 81

Not on the map. Why?

Cited by 56 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Task-oriented dialogue (TOD) systems have been widely used by mobile phone intelligent assistants to accomplish tasks such as calendar scheduling or hotel reservation. Current TOD systems usually focus on multi-turn text/speech interaction, then they would call back-end APIs designed for TODs to perform the task. However, this API-based architecture greatly limits the information-searching capability of intelligent assistants and may even lead to task failure if TOD-specific APIs are not available or the task is too complicated to be executed by the provided APIs. In this paper, we propose a new TOD architecture: GUI-based task-oriented dialogue system (GUI-TOD). A GUI-TOD system can directly perform GUI operations on real APPs and execute tasks without invoking TOD-specific backend APIs. Furthermore, we release META-GUI, a dataset for training a Multi-modal convErsaTional Agent on mobile GUI. We also propose a multi-model action prediction and response model, which show promising results on META-GUI. The dataset, codes and leaderboard are publicly available.

Key facts. Cited by 81 papers in the map (data through 14 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

30 of 42 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  6. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  7. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  8. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  9. 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
  10. 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
  11. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  12. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
18 more
  1. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 95
  2. 2020Recent Advances and Challenges in Task-oriented Dialog Systemcited by 78
  3. 2020VD-BERT: A Unified Vision and Dialog Transformer with BERTcited by 55
  4. 2020ActionBert: Leveraging User Actions for Semantic Understanding of User Interfacescited by 43
  5. 2021Grounding Open-Domain Instructions to Automate Web Support Taskscited by 43
  6. 2019Multi-step Reasoning via Recurrent Dual Attention for Visual Dialogcited by 41
  7. 2018Towards Universal Dialogue State Trackingcited by 33
  8. 2020History for Visual Dialog: Do we really need it?cited by 32
  9. 2020Efficient Context and Schema Fusion Networks for Multi-Domain Dialogue State Trackingcited by 27
  10. 2019Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrationscited by 26
  11. 2018Mapping Natural Language Commands to Web Elementscited by 23
  12. 2020BiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialoguescited by 22
  13. 2020FLIN: A Flexible Natural Language Interface for Web Navigationcited by 18
  14. 2020Video-Grounded Dialogues with Pretrained Generation Language Modelscited by 16
  15. 2019AgentGraph: Towards Universal Dialogue Management with Structured Deep Reinforcement Learningcited by 12
  16. 2020Distributed Structured Actor-Critic Reinforcement Learning for Universal Dialogue Managementcited by 12
  17. 2021Large-Scale Modeling of Mobile User Click Behaviors Using Deep Learningcited by 1
  18. 2021Few-Shot NLU with Vector Projection Distance and Abstract Triangular CRFcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 26; 2025: 36; 2026: 10.

81 papers.

Sort:

By area:

  1. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  3. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  4. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  5. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  6. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  7. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 184
  8. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  9. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  10. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  11. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  12. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 114
38 more
  1. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agentscited by 107
  2. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 103
  3. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 102
  4. 2024Large Language Models Empowered Personalized Web Agentscited by 66
  5. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
  6. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
  7. 2023Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automationcited by 58
  8. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 56
  9. 2024Benchmarking Mobile Device Control Agents across Diverse Configurationscited by 55
  10. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandingcited by 48
  11. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  12. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  13. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  14. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 39
  15. 2023Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interactioncited by 36
  16. 2024LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automationcited by 36
  17. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 35
  18. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  19. 2024AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generationcited by 34
  20. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
  21. 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agentscited by 32
  22. 2025OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agentscited by 25
  23. 2024Dual-View Visual Contextualization for Web Navigationcited by 24
  24. 2023WebVLN: Vision-and-Language Navigation on Websitescited by 19
  25. 2024MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domainscited by 19
  26. 2025Morae: Proactively Pausing UI Agents for User Choicescited by 19
  27. 2024Tur[k]ingBench: A Challenge Benchmark for Web Agentscited by 18
  28. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  29. 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
  30. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  31. 2025GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomaliescited by 15
  32. 2024MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automationcited by 14
  33. 2025Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigationcited by 13
  34. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  35. 2025Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling Systemcited by 12
  36. 2025OpenApps: Simulating Environment Variations to Measure UI-Agent Reliabilitycited by 12
  37. 2025Does Chain-of-Thought Reasoning Help Mobile GUI Agent? An Empirical Studycited by 11
  38. 2025AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Managementcited by 11

50 most cited of 81 All 81 (JSON)