Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen?
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
Recent advances in mobile agents are dominated by the GUI paradigm, in which agents perceive UI information and emit screen interactions. However, mobile platforms also expose a command-line interface (CLI) that provides direct access to device services and data. We argue CLI deserves first-class consideration alongside GUI. We evaluate three coding agents (Claude Code, Terminus-2, mini-swe-agent) across four model APIs on AndroidWorld and MobileWorld without any mobile-specific post-training, comparing against three reproducible GUI baselines (GUI-Owl-1.5-32B, MAI-UI, Qwen3-VL-32B). Claude Code (Opus 4.7) reaches 71.8% and 51.9%, outperforming every reproducible GUI baseline (69.3/68.1/57.8% on AndroidWorld; 43.2/26.3/13.3% on MobileWorld), while every other CLI configuration remains competitive. To establish the paradigm's ceiling, we provide oracle CLI solutions that reach 88.8% on AndroidWorld (103/116 tasks CLI-solvable) and 86.3% on MobileWorld (101/117 tasks CLI-solvable), indicating substantial room for future improvement. To cover everyday user intents beyond the GUI scope, we introduce the CLI-Advantage Task Suite, comprising 45 templates across five categories: bulk operations, multi-condition filtering, aggregation, cross-app workflows, and hidden device state. Every CLI agent outperforms every GUI baseline in all five categories, with substantially fewer steps per task (10.7 vs. 18.6). To support future research on mobile CLI agents, we will open-source agent implementations, oracle solutions, the CLI-Advantage suite, and evaluation infrastructure.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 81 references that are in the map. First posted on arXiv 16 Jun 2026.
Builds on
30 of 81 references.
Closest ancestors · 19
Same direction, newest first
- 2026Generalization in Online Reinforcement Learning for Mobile Agents8 Mar
- 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents15 Feb
- 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec
- 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments22 Dec
- 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents10 Sep
- 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2 Sep
- 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation21 Aug
- 2025UI-Venus Technical Report: Building High-performance UI Agents with RFT14 Aug
- 2025Agent-SAMA: State-Aware Mobile Assistant29 May
- 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Survey4 Nov
- 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents31 Oct
- 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation19 Oct
7 more
- 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
- 2024MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents12 Jun
- 2024On the Effects of Data Scale on UI Control Agents6 Jun
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
- 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
- 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
Other references · 11
- 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
- 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
- 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
- 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 248
- 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
- 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
- 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
- 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
- 2026SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?cited by 8
- 2026The Tool Illusion: Rethinking Tool Use in Web Agentscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents
- 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
- 2026VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics
- 2025Mobile GUI Agents under Real-world Threats: Are We There Yet?
- 2024MobileViews: A Million-scale and Diverse Mobile GUI Dataset
- 2026CLI-Anything: Towards Agent-Native Computer Use
- 2026MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environment
- 2026PalmClaw: A Native On-Device Agent Framework for Mobile Phones