# FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents

arXiv: https://arxiv.org/abs/2507.21071 · HTML: https://mlsanity.com/paper/2507.21071/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 33 of its 41 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 30 of 33)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://mlsanity.com/paper/2412.10302/) (2024)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [AppAgent: Multimodal Agents as Smartphone Users](https://mlsanity.com/paper/2312.13771/) (2023)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis](https://mlsanity.com/paper/2307.12856/) (2023)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction](https://mlsanity.com/paper/2412.04454/) (2024)
- [On the Effects of Data Scale on UI Control Agents](https://mlsanity.com/paper/2406.03679/) (2024)
- [Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control](https://mlsanity.com/paper/2306.07863/) (2023)
- [AutoDroid: LLM-powered Task Automation in Android](https://mlsanity.com/paper/2308.15272/) (2023)
- [AgentSquare: Automatic LLM Agent Search in Modular Design Space](https://mlsanity.com/paper/2410.06153/) (2024)
- [GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation](https://mlsanity.com/paper/2311.07562/) (2023)
- [AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents](https://mlsanity.com/paper/2407.17490/) (2024)
- [AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents](https://mlsanity.com/paper/2410.24024/) (2024)
- [Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance](https://mlsanity.com/paper/2410.12361/) (2024)
- [SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation](https://mlsanity.com/paper/2410.15164/) (2024)
- [Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents](https://mlsanity.com/paper/2402.09205/) (2024)
- [Understanding the Weakness of Large Language Model Agents within a Complex Android Environment](https://mlsanity.com/paper/2402.06596/) (2024)
- [Benchmarking Mobile Device Control Agents across Diverse Configurations](https://mlsanity.com/paper/2404.16660/) (2024)
- [Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions](https://mlsanity.com/paper/2408.04168/) (2024)
- [Division-of-Thoughts: Harnessing Hybrid Language Model Synergy for Efficient On-Device Agents](https://mlsanity.com/paper/2502.04392/) (2025)
- [AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems](https://mlsanity.com/paper/2505.19623/) (2025)

## Built on by (most cited first, 20 of 20)

- [A Survey on Agentic Multimodal Large Language Models](https://mlsanity.com/paper/2510.10991/) (2025)
- [KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation](https://mlsanity.com/paper/2604.08455/) (2026)
- [PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Records](https://mlsanity.com/paper/2601.09636/) (2026)
- [ProAgentBench: Evaluating LLM Agents for Proactive Assistance with Real-World Data](https://mlsanity.com/paper/2602.04482/) (2026)
- [ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems in the Wild](https://mlsanity.com/paper/2512.06721/) (2025)
- [Me-Agent: A Personalized Mobile Agent with Two-Level User Habit Learning for Enhanced Interaction](https://mlsanity.com/paper/2601.20162/) (2026)
- [ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices](https://mlsanity.com/paper/2602.21858/) (2026)
- [Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Models](https://mlsanity.com/paper/2602.02039/) (2026)
- [PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent](https://mlsanity.com/paper/2603.29318/) (2026)
- [TriggerBench: Investigating Prospective Memory for Large Language Models](https://mlsanity.com/paper/2606.23459/) (2026)
- [GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks](https://mlsanity.com/paper/2603.25864/) (2026)
- [Do Proactive Agents Really Need an LLM to Decide When to Wake and What to Anchor?](https://mlsanity.com/paper/2605.30152/) (2026)
- [ProEvent: An Event-centric Benchmark for Proactive Agents](https://mlsanity.com/paper/2607.17701/) (2026)
- [TIDE: Proactive Multi-Problem Discovery via Template-Guided Iteration](https://mlsanity.com/paper/2606.04743/) (2026)
- [Beyond Binary: Reframing GUI Critique as Continuous Semantic Alignment](https://mlsanity.com/paper/2605.14311/) (2026)
- [SocialPersona: Benchmarking Personalized Profiling and Response with Multimodal Social-Media Context](https://mlsanity.com/paper/2606.26654/) (2026)
- [UserToolBench: A User-Profile-Hidden Benchmark for Personalized Decision Making in Tool-Use LLMs](https://mlsanity.com/paper/2608.10042/) (2026)
- [PastForward: Faster On-Device GUI Agents via Computational Experience Reuse](https://mlsanity.com/paper/2609.32166/) (2026)
- [Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents](https://mlsanity.com/paper/2609.33304/) (2026)
- [Who Is Your Agent Serving? Provider-Side Indirect Prompt Injection in Proactive Agents](https://mlsanity.com/paper/2610.05266/) (2026)

## Abstract (arXiv)

Mobile GUI agents are becoming critical tools to improve user experience on smart devices, with multimodal large language models (MLLMs) emerging as the dominant paradigms in this domain. Current agents, however, rely on explicit human instructions, overlooking the potential to leverage the contextual information (like location, time, user profile) and historical data for proactive task suggestions. Besides, previous works focus on optimizing the success rate during task execution, but pay less attention to the personalized execution trajectory, thereby neglecting potentially vast differences in user preferences. To address these challenges, we introduce the FingerTip 20K benchmark. We collected 20K unique human demonstrations of multi-step Android device interactions across a variety of everyday apps. These demonstrations are not isolated but are continuously acquired from the users' long-term usage in their real lives, and encompass essential user-related contextual information. The benchmark contains two new tracks: proactive task suggestions by analyzing environment observation and users' previous intents, and personalized task execution by catering to users' action preferences. Our experiments reveal that the tracks we propose pose significant challenges for leveraging user-related information in GUI tasks. We also performed a human study to show that there exists a huge gap between existing agents and humans. The model fine-tuned with the data we collected effectively utilized user information and achieved good results, highlighting the potential of our approach in building more user-oriented mobile LLM agents. Our code is open-source at https://github.com/tsinghua-fib-lab/FingerTip-20K for reproducibility.