# Towards Automating Data Access Permissions in AI Agents

arXiv: https://arxiv.org/abs/2511.17959 · HTML: https://mlsanity.com/paper/2511.17959/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 88 references that are in the map.
- First posted on arXiv 22 Nov 2025.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [Defeating Prompt Injections by Design](https://mlsanity.com/paper/2503.18813/) (2025)
- [AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents](https://mlsanity.com/paper/2406.13352/) (2024)
- [IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems](https://mlsanity.com/paper/2403.04960/) (2024)
- [We're Afraid Language Models Aren't Modeling Ambiguity](https://mlsanity.com/paper/2304.14399/) (2023)
- [AirGapAgent: Protecting Privacy-Conscious Conversational Agents](https://mlsanity.com/paper/2405.05175/) (2024)
- [LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI's ChatGPT Plugins](https://mlsanity.com/paper/2309.10254/) (2023)
- [ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluation](https://mlsanity.com/paper/2405.14125/) (2024)
- [Annotation alignment: Comparing LLM and human annotations of conversational safety](https://mlsanity.com/paper/2406.06369/) (2024)
- [Confidence in the Reasoning of Large Language Models](https://mlsanity.com/paper/2412.15296/) (2024)
- [Dissecting Human and LLM Preferences](https://mlsanity.com/paper/2402.11296/) (2024)
- [Android Permissions Remystified: A Field Study on Contextual Integrity](https://mlsanity.com/paper/1504.03747/) (2015)
- [An In-Depth Investigation of Data Collection in LLM App Ecosystems](https://mlsanity.com/paper/2408.13247/) (2024)
- [LLM App Store Analysis: A Vision and Roadmap](https://mlsanity.com/paper/2404.12737/) (2024)
- [Tracking, Profiling, and Ad Targeting in the Alexa Echo Smart Speaker Ecosystem](https://mlsanity.com/paper/2204.10920/) (2022)
- [OVRseen: Auditing Network Traffic and Privacy Policies in Oculus VR](https://mlsanity.com/paper/2106.05407/) (2021)
- [Predicting Privacy Preferences for Smart Devices as Norms](https://mlsanity.com/paper/2302.10650/) (2023)

## Built on by (most cited first, 21 of 21)

- [The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey](https://mlsanity.com/paper/2603.11088/) (2026)
- [ceLLMate: Sandboxing Browser AI Agents](https://mlsanity.com/paper/2512.12594/) (2025)
- [Reframing LLM Agent Security as an Agent-Human Interaction Problem](https://mlsanity.com/paper/2605.24309/) (2026)
- [Behavioral Integrity Verification for AI Agent Skills](https://mlsanity.com/paper/2605.11770/) (2026)
- [Can LLMs Make (Personalized) Access Control Decisions?](https://mlsanity.com/paper/2511.20284/) (2025)
- [Personalizing Agent Privacy Decisions via Logical Entailment](https://mlsanity.com/paper/2512.05065/) (2025)
- [How Agents Ask for Permission: User Permissions for AI Agents, from Interfaces to Enforcement](https://mlsanity.com/paper/2607.13718/) (2026)
- [Janus: a Playground for User-Involved Agentic Permission Management](https://mlsanity.com/paper/2607.01510/) (2026)
- [AgentFlow: A Flow-Centric Policy Language and Framework for Securing LLM Agent Systems](https://mlsanity.com/paper/2608.22868/) (2026)
- [MiniScope: A Least Privilege Framework for Authorizing Tool Calling Agents](https://mlsanity.com/paper/2512.11147/) (2025)
- [PrivScope: Task-scoped Disclosure Control for Hybrid Agentic Systems](https://mlsanity.com/paper/2605.16630/) (2026)
- [SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents](https://mlsanity.com/paper/2606.02302/) (2026)
- [Agent Data Injection Attacks are Realistic Threats to AI Agents](https://mlsanity.com/paper/2607.05120/) (2026)
- [Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent](https://mlsanity.com/paper/2607.18257/) (2026)
- [PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior](https://mlsanity.com/paper/2605.12147/) (2026)
- ["Allow" to Achieve, Over-Privileged Inadvertently: The Unintended Cost of Task-Completion-Driven Pop-up Decisions in Mobile GUI Agents](https://mlsanity.com/paper/2608.04755/) (2026)
- [Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach?](https://mlsanity.com/paper/2608.27443/) (2026)
- [Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks](https://mlsanity.com/paper/2609.17817/) (2026)
- [Ajar: Measuring Open Privilege in Agent Defenses](https://mlsanity.com/paper/2609.26900/) (2026)
- [When Valid Tool Calls Change Meaning: Formation-Consistent Dispatch for LLM Agents](https://mlsanity.com/paper/2609.35088/) (2026)
- [Pincer: Resource Authorization for Agents using a Digital Twin](https://mlsanity.com/paper/2610.02569/) (2026)

## Abstract (arXiv)

As AI agents attempt to autonomously act on users' behalf, they raise transparency and control issues. We argue that permission-based access control is indispensable in providing meaningful control to the users, but conventional permission models are inadequate for the automated agentic execution paradigm. We therefore propose automated permission management for AI agents. Our key idea is to conduct a user study to identify the factors influencing users' permission decisions and to encode these factors into an ML-based permission management assistant capable of predicting users' future decisions. We find that participants' permission decisions are influenced by communication context but importantly individual preferences tend to remain consistent within contexts, and align with those of other participants. Leveraging these insights, we develop a permission prediction model achieving 85.1% accuracy overall and 94.4% for high-confidence predictions. We find that even without using permission history, our model achieves an accuracy of 66.9%, and a slight increase of training samples (i.e., 1-4) can substantially increase the accuracy by 10.8%.