# Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts

arXiv: https://arxiv.org/abs/2404.02475 · HTML: https://mlsanity.com/paper/2404.02475/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2024.

## Built on by (most cited first, 8 of 8)

- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models](https://mlsanity.com/paper/2411.05451/) (2024)
- [EcoAgent: An Efficient Device-Cloud Collaborative Multi-Agent Framework for Mobile Automation](https://mlsanity.com/paper/2505.05440/) (2025)
- [Characterizing Unintended Consequences in Human-GUI Agent Collaboration for Web Browsing](https://mlsanity.com/paper/2505.09875/) (2025)
- [UserCentrix: An Agentic Memory-augmented AI Framework for Smart Spaces](https://mlsanity.com/paper/2505.00472/) (2025)
- [DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfaces](https://mlsanity.com/paper/2509.13444/) (2025)
- [Image First or Text First? Optimising the Sequencing of Modalities in Large Language Model Prompting and Reasoning Tasks](https://mlsanity.com/paper/2410.03062/) (2024)
- [Software Engineering for and with GUI Agent](https://mlsanity.com/paper/2608.09278/) (2026)

## Abstract (arXiv)

UI task automation enables efficient task execution by simulating human interactions with graphical user interfaces (GUIs), without modifying the existing application code. However, its broader adoption is constrained by the need for expertise in both scripting languages and workflow design. To address this challenge, we present Prompt2Task, a system designed to comprehend various task-related textual prompts (e.g., goals, procedures), thereby generating and performing the corresponding automation tasks. Prompt2Task incorporates a suite of intelligent agents that mimic human cognitive functions, specializing in interpreting user intent, managing external information for task generation, and executing operations on smartphones. The agents can learn from user feedback and continuously improve their performance based on the accumulated knowledge. Experimental results indicated a performance jump from a 22.28% success rate in the baseline to 95.24% with Prompt2Task, requiring an average of 0.69 user interventions for each new task. Prompt2Task presents promising applications in fields such as tutorial creation, smart assistance, and customer service.