Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization
On the map: LLM agents and tool use → GUI and mobile agents
also in 1 more
Abstract (arXiv)
Mobile GUI agents powered by Multimodal Large Language Models (MLLMs) can execute complex tasks on mobile devices. Despite this progress, most existing systems still optimize task success or efficiency, neglecting users' privacy personalization. In this paper, we study the often-overlooked problem of agent personalization. We observe that personalization can induce systematic structural heterogeneity in execution trajectories. For example, privacy-first users often prefer protective actions, e.g., refusing permissions, logging out, and minimizing exposure, leading to logically different execution trajectories from utility-first users. Such variable-length and structurally different trajectories make standard preference optimization unstable and less informative. To address this issue, we propose Trajectory Induced Preference Optimization (TIPO), which uses preference-intensity weighting to emphasize key privacy-related steps and padding gating to suppress alignment noise. Results on our Privacy Preference Dataset show that TIPO improves persona alignment and distinction while preserving strong task executability, achieving 65.60% SR, 46.22 Compliance, and 66.67% PD, outperforming existing optimization methods across various GUI tasks. The code and dataset will be publicly released at https://github.com/Zhixin-L/TIPO.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 26 of its 42 references that are in the map. First posted on arXiv 13 Apr 2026.
Builds on
26 of 42 references.
Closest ancestors · 10
Same direction, newest first
- 2026AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents19 Mar
- 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation21 Aug
- 2025Towards Trustworthy GUI Agents: A Survey30 Mar
- 2025A Survey on (M)LLM-Based GUI Agents27 Mar
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2024GUI Agents: A Survey18 Dec
- 2024Large Language Model-Brained GUI Agents: A Survey27 Nov
- 2024GUI Agents with Foundation Models: A Comprehensive Survey7 Nov
- 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation19 Oct
- 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactions5 Aug
Other references · 16
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
- 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
- 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
- 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
- 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
- 2024Personalized Language Modeling from Personalized Human Feedbackcited by 124
- 2025PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistantscited by 44
- 2025PROPER: A Progressive Learning Framework for Personalized Large Language Models with Group-Level Adaptationcited by 22
- 2026Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User Historycited by 14
4 more
- 2025Mind the Third Eye! Benchmarking Privacy Awareness in MLLM-powered Smartphone Agentscited by 8
- 2025Explicit Preference Optimization: No Need for an Implicit Reward Modelcited by 6
- 2026Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisiblecited by 5
- 2018Privacy attitudes and concerns in the digital lives of older adults: Westin's privacy attitude typology revisitedcited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026PriMobiBench: Characterizing Visual Privacy Leakage in VLM-Driven Mobile GUI Agents
- 2026GUIGuard-Bench: Toward a General Evaluation for Privacy-Preserving GUI Agents
- 2026KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation
- 2026PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent
- 2026PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Records
- 2026Do Phone-Use Agents Respect Your Privacy?
- 2026Synthetic Interaction Data for Scalable Personalization in Large Language Models
- 2025FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents