mlsanityData to 14 Oct 2026

Cradle: Empowering Foundation Agents Towards General Computer Control

arXiv 2403.03186 ↗First posted 5 Mar 2024cs.AIcited by 117

Not on the map. Why?

Cited by 76 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite the success in specific scenarios, existing foundation agents still struggle to generalize across various virtual scenarios, mainly due to the dramatically different encapsulations of environments with manually designed observation and action spaces. To handle this issue, we propose the General Computer Control (GCC) setting to restrict foundation agents to interact with software through the most unified and standardized interface, i.e., using screenshots as input and keyboard and mouse actions as output. We introduce Cradle, a modular and flexible LMM-powered framework, as a preliminary attempt towards GCC. Enhanced by six key modules, Cradle can understand input screenshots and output executable code for low-level keyboard and mouse control after high-level planning, so that Cradle can interact with any software and complete long-horizon complex tasks without relying on any built-in APIs. Experimental results show that Cradle exhibits remarkable generalizability and impressive performance across four previously unexplored commercial video games, five software applications, and a comprehensive benchmark, OSWorld. Cradle is the first to enable foundation agents to follow the main storyline and complete 40-minute-long real missions in the complex AAA game Red Dead Redemption 2 (RDR2). Cradle can also create a city of a thousand people in Cities: Skylines, farm and harvest parsnips in Stardew Valley, and trade and bargain with a maximal weekly total profit of 87% in Dealer's Life 2. Cradle can not only operate daily software, like Chrome, Outlook, and Feishu, but also edit images and videos using Meitu and CapCut. Cradle greatly extends the reach of foundation agents by enabling the easy conversion of any software, especially complex games, into benchmarks to evaluate agents' various abilities and facilitate further data collection, thus paving the way for generalist agents.

Key facts. Cited by 117 papers in the map (data through 14 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

35 of 47 references.

References · 35

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  7. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  8. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  9. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  11. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  12. 2019The StarCraft Multi-Agent Challengecited by 597
23 more
  1. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  2. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  3. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  4. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  5. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  6. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  7. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  8. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  9. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  10. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  11. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  12. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  13. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 205
  14. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 187
  15. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 171
  16. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  17. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
  18. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  19. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 102
  20. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 96
  21. 2023Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approachcited by 79
  22. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  23. 2024CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agentscited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 54; 2026: 51.

117 papers.

Sort:

By area:

  1. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  2. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  4. 2026SoK: Agentic Skills -- Beyond Tool Use in LLM Agentscited by 130
  5. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 103
  6. 2024GUI Agents: A Surveycited by 103
  7. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 93
  8. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  9. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
  10. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 55
  11. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 49
  12. 2026Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agentscited by 47
38 more
  1. 2026Code as Agent Harnesscited by 47
  2. 2025AgentOrchestra: Orchestrating Multi-Agent Intelligence with the Tool-Environment-Agent(TEA) Protocolcited by 39
  3. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  4. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  5. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 34
  6. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
  7. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  8. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 31
  9. 2025G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learningcited by 29
  10. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflowcited by 28
  11. 2026GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agentscited by 28
  12. 2026NitroGen: An Open Foundation Model for Generalist Gaming Agentscited by 27
  13. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 25
  14. 2025Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agentscited by 24
  15. 2024A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challengescited by 22
  16. 2025Being-0: A Humanoid Robotic Agent with Vision-Language Models and Modular Skillscited by 22
  17. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
  18. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 21
  19. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Controlcited by 20
  20. 2025Are Large Vision Language Models Good Game Players?cited by 19
  21. 2025Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learningcited by 18
  22. 2026A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applicationscited by 18
  23. 2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systemscited by 16
  24. 2025RoboMemory: A Brain-inspired Multi-memory Agentic Framework for Interactive Environmental Learning in Physical Embodied Systemscited by 15
  25. 2025Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agentscited by 13
  26. 2025Optimus-3: Dual-Router Aligned Mixture-of-Experts Agent with Dual-Granularity Reasoning-Aware Policy Optimizationcited by 13
  27. 2025VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraftcited by 13
  28. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
  29. 2025Prune4Web: DOM Tree Pruning Programming for Web Agentcited by 12
  30. 2024MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agentscited by 11
  31. 2025V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Modelscited by 11
  32. 2025General Modular Harness for LLM Agents in Multi-Turn Gaming Environmentscited by 11
  33. 2025MapAgent: Trajectory-Constructed Memory-Augmented Planning for Mobile Task Automationcited by 11
  34. 2025AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Managementcited by 11
  35. 2026TRACER: Trajectory Risk Aggregation for Critical Episodes in Agentic Reasoningcited by 11
  36. 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agentscited by 11
  37. 2024LLM-Based World Models Can Make Decisions Solely, But Rigorous Evaluations are Neededcited by 10
  38. 2024SmartAgent: Chain-of-User-Thought for Embodied Personalized Agent in Cyber Worldcited by 10

50 most cited of 117 All 117 (JSON)