# GPTVoiceTasker: Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learning

arXiv: https://arxiv.org/abs/2401.14268 · HTML: https://mlsanity.com/paper/2401.14268/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jan 2024.

## Built on by (most cited first, 14 of 14)

- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection](https://mlsanity.com/paper/2501.04575/) (2025)
- [From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts](https://mlsanity.com/paper/2410.09006/) (2024)
- [DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues](https://mlsanity.com/paper/2506.22853/) (2025)
- [CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents](https://mlsanity.com/paper/2603.10577/) (2026)
- [A Multimodal GUI Architecture for Interfacing with LLM-Based Conversational Assistants](https://mlsanity.com/paper/2510.06223/) (2025)
- [Software Engineering for and with GUI Agent](https://mlsanity.com/paper/2608.09278/) (2026)
- [From Exploration to Revelation: Detecting Dark Patterns in Mobile Apps](https://mlsanity.com/paper/2411.18084/) (2024)
- [From User Interface to Agent Interface: Efficiency Optimization of UI Representations for LLM Agents](https://mlsanity.com/paper/2512.13438/) (2025)
- [SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesis](https://mlsanity.com/paper/2601.18305/) (2026)
- [Insight: Enhancing Mobile Accessibility for Blind and Visually Impaired Users with LLMs](https://mlsanity.com/paper/2605.09803/) (2026)
- [Fishing Out Free Riders: Shapley-Based Reward Attribution for Parallel Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2607.18979/) (2026)
- [SenWorld: A Digital-Twin Simulation for Generating Context-Rich Evaluation Data](https://mlsanity.com/paper/2607.19949/) (2026)
- [VoCa: Designing Speech-Canvas Interaction for Voice-Based Conversational Agents](https://mlsanity.com/paper/2610.04706/) (2026)

## Abstract (arXiv)

Virtual assistants have the potential to play an important role in helping users achieves different tasks. However, these systems face challenges in their real-world usability, characterized by inefficiency and struggles in grasping user intentions. Leveraging recent advances in Large Language Models (LLMs), we introduce GptVoiceTasker, a virtual assistant poised to enhance user experiences and task efficiency on mobile devices. GptVoiceTasker excels at intelligently deciphering user commands and executing relevant device interactions to streamline task completion. The system continually learns from historical user commands to automate subsequent usages, further enhancing execution efficiency. Our experiments affirm GptVoiceTasker's exceptional command interpretation abilities and the precision of its task automation module. In our user study, GptVoiceTasker boosted task efficiency in real-world scenarios by 34.85%, accompanied by positive participant feedback. We made GptVoiceTasker open-source, inviting further research into LLMs utilization for diverse tasks through prompt engineering and leveraging user usage data to improve efficiency.