mlsanityData to 7 Oct 2026

Android in the Zoo: Chain-of-Action-Thought for GUI Agents

arXiv 2403.02713 ↗First posted 5 Mar 2024cs.CL cs.CV cs.HCcited by 156

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Large language model (LLM) leads to a surge of autonomous GUI agents for smartphone, which completes a task triggered by natural language through predicting a sequence of actions of API. Even though the task highly relies on past actions and visual observations, existing studies typically consider little semantic information carried out by intermediate screenshots and screen operations. To address this, this work presents Chain-of-Action-Thought (dubbed CoAT), which takes the description of the previous actions, the current screen, and more importantly the action thinking of what actions should be performed and the outcomes led by the chosen action. We demonstrate that, in a zero-shot setting upon three off-the-shelf LMMs, CoAT significantly improves the action prediction compared to previous proposed context modeling. To further facilitate the research in this line, we construct a dataset Android-In-The-Zoo (AitZ), which contains 18,643 screen-action pairs together with chain-of-action-thought annotations. Experiments show that fine-tuning a 1B model (i.e. AUTO-UI-base) on our AitZ dataset achieves on-par performance with CogAgent-Chat-18B.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 51 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

28 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
  2. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  8. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  9. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  10. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  11. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  12. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
14 more
  1. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  2. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  3. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  4. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  5. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  6. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  7. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  8. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  9. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  10. 2023Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automationcited by 57
  11. 2021Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environmentscited by 28
  12. 2022UGIF: UI Grounded Instruction Followingcited by 25
  13. 2022Towards Better Semantic Understanding of Mobile Interfacescited by 24
  14. 2023Applying Learning-from-observation to household service robots: three common-sense formulationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 89; 2026: 37.

156 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  3. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  4. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  5. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  6. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  7. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  8. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  9. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  10. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  11. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  12. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
38 more
  1. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  2. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  3. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  4. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  5. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
  6. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  7. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  8. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  9. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
  10. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
  11. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  12. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  13. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
  14. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  15. 2024TinyAgent: Function Calling at the Edgecited by 55
  16. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 51
  17. 2024Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMscited by 48
  18. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandingcited by 46
  19. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  20. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  21. 2024VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Taskscited by 43
  22. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 38
  23. 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
  24. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  25. 2025ViMo: A Generative Visual GUI World Model for App Agentscited by 33
  26. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  27. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  28. 2025Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Modelscited by 30
  29. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 29
  30. 2025MedBrowseComp: Benchmarking Medical Deep Research and Computer Usecited by 28
  31. 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
  32. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  33. 2025OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agentscited by 25
  34. 2025BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanismcited by 25
  35. 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 25
  36. 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
  37. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 22
  38. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Syntax: Action Semantics Learning for App Agents
  2. 2024MobileViews: A Million-scale and Diverse Mobile GUI Dataset
  3. 2025AppVLM: A Lightweight Vision Language Model for Online App Control
  4. 2025ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction
  5. 2026MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization
  6. 2026PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents
  7. 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agents
  8. 2026UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents