mlsanityData to 7 Oct 2026

AutoGLM: Autonomous Foundation Agents for GUIs

arXiv 2411.00820 ↗First posted 28 Oct 2024cs.HC cs.AI cs.CLcited by 96

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

We present AutoGLM, a new series in the ChatGLM family, designed to serve as foundation agents for autonomous control of digital devices through Graphical User Interfaces (GUIs). While foundation models excel at acquiring human knowledge, they often struggle with decision-making in dynamic real-world environments, limiting their progress toward artificial general intelligence. This limitation underscores the importance of developing foundation agents capable of learning through autonomous environmental interactions by reinforcing existing models. Focusing on Web Browser and Phone as representative GUI scenarios, we have developed AutoGLM as a practical foundation agent system for real-world GUI interactions. Our approach integrates a comprehensive suite of techniques and infrastructures to create deployable agent systems suitable for user delivery. Through this development, we have derived two key insights: First, the design of an appropriate "intermediate interface" for GUI control is crucial, enabling the separation of planning and grounding behaviors, which require distinct optimization for flexibility and accuracy respectively. Second, we have developed a novel progressive training framework that enables self-evolving online curriculum reinforcement learning for AutoGLM. Our evaluations demonstrate AutoGLM's effectiveness across multiple domains. For web browsing, AutoGLM achieves a 55.2% success rate on VAB-WebArena-Lite (improving to 59.1% with a second attempt) and 96.2% on OpenTable evaluation tasks. In Android device control, AutoGLM attains a 36.2% success rate on AndroidLab (VAB-Mobile) and 89.7% on common tasks in popular Chinese APPs.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 36 of its 43 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

36 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  2. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
  3. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
  4. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
20 more
  1. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  2. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  5. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  6. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  7. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  8. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  9. 2024Agent Workflow Memorycited by 342
  10. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  11. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  12. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  13. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  14. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  15. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  16. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 134
  17. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 101
  18. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 100
  19. 2024Planning In Natural Language Improves LLM Search For Code Generationcited by 79
  20. 2024NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Promptscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 41; 2026: 52.

96 papers.

Sort:

By area:

  1. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  3. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 128
  4. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  5. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 73
  6. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 72
  7. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  8. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  9. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 49
  10. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  11. 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Userscited by 44
  12. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
38 more
  1. 2026Code as Agent Harnesscited by 40
  2. 2026UI-Venus-1.5 Technical Reportcited by 39
  3. 2025Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deploymentcited by 38
  4. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  5. 2024AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generationcited by 31
  6. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  7. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  8. 2025OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulationcited by 26
  9. 2025OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agentscited by 25
  10. 2025BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanismcited by 25
  11. 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curationcited by 24
  12. 2025OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflowscited by 18
  13. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  14. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  15. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 14
  16. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  17. 2025EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Explorationcited by 13
  18. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
  19. 2026The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agentscited by 12
  20. 2025Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agentscited by 11
  21. 2025Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling Systemcited by 11
  22. 2025From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agentscited by 10
  23. 2025Context manipulation attacks : Web agents are susceptible to corrupted memorycited by 10
  24. 2026ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agentscited by 10
  25. 2025MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learningcited by 9
  26. 2025GEM: Gaussian Embedding Modeling for Out-of-Distribution Detection in GUI Agentscited by 8
  27. 2026Mapping the Design Space of User Experience for Computer Use Agentscited by 7
  28. 2026AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wildcited by 7
  29. 2025OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?cited by 6
  30. 2025CRAFT-GUI: Curriculum-Reinforced Agent For GUI Taskscited by 6
  31. 2026Generalization in Online Reinforcement Learning for Mobile Agentscited by 6
  32. 2025BTL-UI: Blink-Think-Link Reasoning Model for GUI Agentcited by 5
  33. 2026Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisiblecited by 5
  34. 2026VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnosticscited by 5
  35. 2026CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automationcited by 5
  36. 2026Xiaomi-GUI-0 Technical Reportcited by 5
  37. 2025CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMscited by 4
  38. 2025OpenPhone: Mobile Agentic Foundation Modelscited by 4

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024GUI Agents with Foundation Models: A Comprehensive Survey
  2. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning
  3. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning
  4. 2025Co-EPG: A Framework for Co-Evolution of Planning and Grounding in Autonomous GUI Agents
  5. 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
  6. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitants
  7. 2024GUI Agents: A Survey
  8. 2025A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learning