# PrivacyGuard: Source-Linked Sanitization and Locally Resolvable PII References for Cloud GUI Agents

arXiv: https://arxiv.org/abs/2610.16778 · HTML: https://mlsanity.com/paper/2610.16778/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 37 references that are in the map.
- First posted on arXiv 12 Oct 2026.

## Placed in

- LLM agents and tool use

## Builds on (most cited first, 30 of 33)

- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [AppAgent: Multimodal Agents as Smartphone Users](https://mlsanity.com/paper/2312.13771/) (2023)
- [Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception](https://mlsanity.com/paper/2401.16158/) (2024)
- [OmniParser for Pure Vision Based GUI Agent](https://mlsanity.com/paper/2408.00203/) (2024)
- [Mobile-Agent-v3: Fundamental Agents for GUI Automation](https://mlsanity.com/paper/2508.15144/) (2025)
- [Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration](https://mlsanity.com/paper/2406.01014/) (2024)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents](https://mlsanity.com/paper/2410.24024/) (2024)
- [ScreenAI: A Vision-Language Model for UI and Infographics Understanding](https://mlsanity.com/paper/2402.04615/) (2024)
- [GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer](https://mlsanity.com/paper/2311.08526/) (2023)
- [Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models](https://mlsanity.com/paper/2305.15594/) (2023)
- [Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents](https://mlsanity.com/paper/2602.16855/) (2026)
- [AppAgent v2: Advanced Agent for Flexible Mobile Interactions](https://mlsanity.com/paper/2408.11824/) (2024)
- [The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injections](https://mlsanity.com/paper/2504.11281/) (2025)
- [AppAgentX: Evolving GUI Agents as Proficient Smartphone Users](https://mlsanity.com/paper/2503.02268/) (2025)
- [Hide and Seek (HaS): A Lightweight Framework for Prompt Privacy Protection](https://mlsanity.com/paper/2309.03057/) (2023)
- [Towards Trustworthy GUI Agents: A Survey](https://mlsanity.com/paper/2503.23434/) (2025)
- [Split-and-Denoise: Protect large language model inference with local differential privacy](https://mlsanity.com/paper/2310.09130/) (2023)
- [PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restoration](https://mlsanity.com/paper/2406.01394/) (2024)
- [PrivWeb: Unobtrusive and Content-aware Privacy Protection For Web Agents](https://mlsanity.com/paper/2509.11939/) (2025)
- [DualTAP: A Dual-Task Adversarial Protector for Mobile MLLM Agents](https://mlsanity.com/paper/2511.13248/) (2025)
- [InferDPT: Privacy-Preserving Inference for Closed-box Large Language Model](https://mlsanity.com/paper/2310.12214/) (2023)
- [Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisible](https://mlsanity.com/paper/2602.10139/) (2026)
- [PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection](https://mlsanity.com/paper/2604.15776/) (2026)
- [CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMs](https://mlsanity.com/paper/2510.15455/) (2025)
- [DiffSTR: Controlled Diffusion Models for Scene Text Removal](https://mlsanity.com/paper/2410.21721/) (2024)
- [WebPII: Benchmarking Visual PII Detection for Computer-Use Agents](https://mlsanity.com/paper/2603.17357/) (2026)
- [A System for Name and Address Parsing with Large Language Models](https://mlsanity.com/paper/2601.18014/) (2026)
- [Improvement in Semantic Address Matching using Natural Language Processing](https://mlsanity.com/paper/2404.11691/) (2024)

## Abstract (arXiv)

Cloud graphical user interface (GUI) agents send instructions and screenshots to remote planners, exposing personally identifiable information (PII). Existing placeholder defenses mask detected spans and reuse exact strings. Fragmented or changed forms can expose PII or break entity continuity across turns. PrivacyGuard addresses this private reference problem with source-linked sanitization (SAN) and a structured reference lifecycle (REF). SAN reconstructs fragmented or overlapping evidence and rewrites the complete source extent. REF links changed appearances to session entities and supports locally resolvable references without returning raw values to the upload path. In five-turn sessions scored against common gold, PrivacyGuard reduces audited source-value leakage by 33.55%--82.60% relative to the closest prior full-pipeline system in all four reader--language combinations. REF raises strict cross-form continuity from 0.04 under exact-value reuse to 0.81. In reference-required Android tasks, verified completion rises from 8/30 and 7/30 to 17/30 and 20/30. On AndroidLab, PrivacyGuard achieves the highest or tied-highest task-success point estimate among protection pipelines for all four planners. On PrivScreen, it reduces fixed-reader field exposure by 62.72%--62.84% relative to the same baseline. PrivacyGuard lowers upload exposure while preserving a locally resolvable task path.