mlsanityData to 7 Oct 2026

CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation

arXiv 2402.11941 ↗First posted 19 Feb 2024cs.CLcited by 70

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) have shown remarkable potential as human-like autonomous language agents to interact with real-world environments, especially for graphical user interface (GUI) automation. However, those GUI agents require comprehensive cognition ability including exhaustive perception and reliable action response. We propose a Comprehensive Cognitive LLM Agent, CoCo-Agent, with two novel approaches, comprehensive environment perception (CEP) and conditional action prediction (CAP), to systematically improve the GUI automation performance. First, CEP facilitates the GUI perception through different aspects and granularity, including screenshots and complementary detailed layouts for the visual channel and historical actions for the textual channel. Second, CAP decomposes the action prediction into sub-problems: action type prediction and action target conditioned on the action type. With our technical design, our agent achieves new state-of-the-art performance on AITW and META-GUI benchmarks, showing promising abilities in realistic scenarios. Code is available at https://github.com/xbmxb/CoCo-Agent.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 39 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

24 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  2. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation13 Nov

Other references · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  12. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
10 more
  1. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  2. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  3. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335
  4. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  5. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  6. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  7. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agentscited by 135
  8. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  9. 2023OtterHD: A High-Resolution Multi-modality Modelcited by 72
  10. 2022Towards Better Semantic Understanding of Mobile Interfacescited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 40; 2026: 17.

70 papers.

Sort:

By area:

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  3. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  4. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  5. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  6. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 34
  7. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  8. 2025A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrievalcited by 31
  9. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  10. 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
  11. 2025OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulationcited by 26
  12. 2025OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agentscited by 25
38 more
  1. 2025BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanismcited by 25
  2. 2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23
  3. 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
  4. 2024Dynamic Planning for LLM-based Graphical User Interface Automationcited by 19
  5. 2025Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learningcited by 19
  6. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  7. 2024SirLLM: Streaming Infinite Retentive LLMcited by 16
  8. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  9. 2025VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agentscited by 15
  10. 2024MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automationcited by 14
  11. 2025SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agentscited by 14
  12. 2024Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuningcited by 13
  13. 2025MP-GUI: Modality Perception with MLLMs for GUI Understandingcited by 12
  14. 2025Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agentscited by 12
  15. 2024ClickAgent: Enhancing UI Location Capabilities of Autonomous Agentscited by 11
  16. 2024GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agentcited by 11
  17. 2025Quick on the Uptake: Eliciting Implicit Intents from Human Demonstrations for Personalized Mobile-Use Agentscited by 11
  18. 2026Adaptive Milestone Reward for GUI Agentscited by 11
  19. 2024SmartAgent: Chain-of-User-Thought for Embodied Personalized Agent in Cyber Worldcited by 10
  20. 2025GEM: Gaussian Embedding Modeling for Out-of-Distribution Detection in GUI Agentscited by 8
  21. 2025TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environmentscited by 8
  22. 2025Mind the Third Eye! Benchmarking Privacy Awareness in MLLM-powered Smartphone Agentscited by 8
  23. 2025MVISU-Bench: Benchmarking Mobile Agents for Real-World Tasks by Multi-App, Vague, Interactive, Single-App and Unethical Instructionscited by 7
  24. 2025Modular Task Decomposition and Dynamic Collaboration in Multi-Agent Systems Driven by Large Language Modelscited by 7
  25. 2026AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wildcited by 7
  26. 2024SELU: Self-Learning Embodied MLLMs in Unknown Environmentscited by 6
  27. 2025LaSM: Layer-wise Scaling Mechanism for Defending Pop-up Attack on GUI Agentscited by 6
  28. 2026Agent-Dice: Disentangling Knowledge Updates via Geometric Consensus for Agent Continual Learningcited by 6
  29. 2025See, Think, Act: Online Shopper Behavior Simulation with VLM Agentscited by 5
  30. 2025Textual-to-Visual Iterative Self-Verification for Slide Generationcited by 4
  31. 2025DeskVision: Large Scale Desktop Region Captioning for Advanced GUI Agentscited by 4
  32. 2025Building a Stable Planner: An Extended Finite State Machine Based Planning Module for Mobile GUI Agentcited by 4
  33. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
  34. 2025CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMscited by 4
  35. 2024AssistantX: An LLM-Powered Proactive Assistant in Collaborative Human-Populated Environmentcited by 3
  36. 2025Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Modelscited by 3
  37. 2025Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbationscited by 3
  38. 2025GUI-Rise: Structured Reasoning and History Summarization for GUI Navigationcited by 3

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning
  2. 2026GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents
  3. 2025UIPro: Unleashing Superior Interaction Capability For GUI Agents
  4. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
  5. 2024GUI Agents with Foundation Models: A Comprehensive Survey
  6. 2025ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction
  7. 2025A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learning
  8. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization